Sep 09 15:18:19.071119 ip-10-0-134-183 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 15:18:19.071134 ip-10-0-134-183 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 15:18:19.071144 ip-10-0-134-183 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 15:18:19.071451 ip-10-0-134-183 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 15:18:29.191028 ip-10-0-134-183 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 15:18:29.191049 ip-10-0-134-183 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot de4f6707cbfc435189d0753445f3c971 --
Sep 09 15:20:09.174418 ip-10-0-134-183 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 15:20:09.551132 ip-10-0-134-183 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 15:20:09.551132 ip-10-0-134-183 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 15:20:09.551132 ip-10-0-134-183 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 15:20:09.551132 ip-10-0-134-183 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 15:20:09.551132 ip-10-0-134-183 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 15:20:09.551784 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.551166    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 15:20:09.556029 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556014    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 15:20:09.556029 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556028    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 15:20:09.556029 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556031    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 15:20:09.556145 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556035    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 15:20:09.556145 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556038    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 15:20:09.556145 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556041    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 15:20:09.556145 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556043    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 15:20:09.556145 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556089    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556156    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556160    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556163    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556166    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556169    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556172    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556174    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556177    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556181    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556184    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556186    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556189    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556192    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556198    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556200    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556203    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556206    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556209    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556212    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 15:20:09.556340 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556215    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556217    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556221    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556226    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556306    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556310    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556314    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556323    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556328    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556332    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556336    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556341    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556345    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556350    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556355    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556360    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556364    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556368    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556372    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556381    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 15:20:09.557077 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556386    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556391    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556395    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556399    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556406    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556412    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556417    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556422    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556426    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556431    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556435    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556440    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556449    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556454    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556460    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556464    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556469    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556474    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556478    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 15:20:09.557714 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556482    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556489    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556493    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556497    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556508    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556512    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556517    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556522    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556526    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556530    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556534    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556539    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556543    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556547    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556551    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556555    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556559    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556568    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556578    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556583    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 15:20:09.558275 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556588    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556593    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556598    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556605    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556610    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556615    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556620    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556632    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556637    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556641    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556645    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556649    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.556654    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558042    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558052    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558055    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558058    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558061    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558064    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 15:20:09.558767 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558066    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558069    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558072    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558074    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558077    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558080    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558082    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558085    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558087    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558090    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558093    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558095    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558098    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558100    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558103    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558106    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558109    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558111    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558113    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558136    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 15:20:09.559264 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558139    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558142    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558145    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558148    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558151    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558154    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558156    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558159    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558162    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558164    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558167    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558171    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558175    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558179    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558182    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558185    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558187    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558191    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558193    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 15:20:09.559762 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558196    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558199    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558203    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558207    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558211    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558214    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558217    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558220    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558223    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558226    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558229    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558232    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558236    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558239    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558243    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558246    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558249    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558252    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558255    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 15:20:09.560239 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558258    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558261    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558264    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558267    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558269    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558272    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558275    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558278    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558280    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558283    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558287    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558290    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558294    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558296    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558299    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558302    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558306    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558308    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558311    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558314    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 15:20:09.560696 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558317    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558320    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558322    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558324    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558327    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558329    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558332    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558335    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558338    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558340    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558343    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558345    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558348    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558350    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558353    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558715    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558720    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558723    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558726    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 15:20:09.561219 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558728    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558731    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558734    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558737    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558740    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558744    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558746    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558749    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558751    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558754    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558757    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558759    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558762    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558764    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558767    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558770    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558772    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558775    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558778    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558781    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 15:20:09.561688 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558784    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558787    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558789    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558792    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558794    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558796    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558799    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558801    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558804    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558806    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558809    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558811    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558813    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558818    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558823    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558826    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558829    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558832    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558834    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558837    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 15:20:09.562197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558839    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558842    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558846    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558849    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558852    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558855    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558857    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558860    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558863    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558865    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558868    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558871    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558874    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558876    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558879    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558881    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558884    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558886    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558888    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 15:20:09.562687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558891    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558893    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558896    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558898    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558901    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558903    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558906    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558909    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558912    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558914    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558917    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558920    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558922    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558927    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558930    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558933    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558936    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558938    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558941    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558943    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 15:20:09.563199 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558946    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558949    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558951    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558953    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558956    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558958    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558962    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558965    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558967    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558970    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558972    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558975    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558977    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558980    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558982    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.558984    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559061    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559098    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559106    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559113    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559134    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 15:20:09.563685 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559139    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559146    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559151    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559154    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559157    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559161    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559164    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559167    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559170    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559173    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559176    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559178    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559181    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559186    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559189    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559191    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559194    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559198    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559202    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559205    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559208    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559211    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559214    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559216    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 15:20:09.564316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559219    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559222    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559225    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559229    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559232    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559235    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559239    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559242    2578 flags.go:64] FLAG: --enable-server="true"
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559244    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559248    2578 flags.go:64] FLAG: --event-burst="100"
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559251    2578 flags.go:64] FLAG: --event-qps="50"
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559254    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559257    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559260    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559264    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559267    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559269    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559272    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559275    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559278    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559281    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559284    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559287    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559290    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559293    2578 flags.go:64] FLAG: --feature-gates=""
Sep 09 15:20:09.564890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559296    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559299    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559303    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559306    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559309    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559312    2578 flags.go:64] FLAG: --help="false"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559315    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-134-183.ec2.internal"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559318    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559321    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559324    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559327    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559330    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559334    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559338    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559341    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559344    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559347    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559350    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559352    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559355    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559358    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559361    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559364    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559367    2578 flags.go:64] FLAG: --lock-file=""
Sep 09 15:20:09.565533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559370    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559372    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559375    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559380    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559383    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559386    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559388    2578 flags.go:64] FLAG: --logging-format="text"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559391    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559394    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559397    2578 flags.go:64] FLAG: --manifest-url=""
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559400    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559404    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559407    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559411    2578 flags.go:64] FLAG: --max-pods="110"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559414    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559417    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559420    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559423    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559426    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559429    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559432    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559440    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559443    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559446    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 15:20:09.566204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559449    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559452    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559458    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559461    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559464    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559467    2578 flags.go:64] FLAG: --port="10250"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559470    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559473    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ee94682027767fea"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559476    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559479    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559482    2578 flags.go:64] FLAG: --register-node="true"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559485    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559488    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559491    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559494    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559497    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559501    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559504    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559506    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559509    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559513    2578 flags.go:64] FLAG: --runonce="false"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559516    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559519    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559522    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559524    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 15:20:09.566827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559527    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559530    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559533    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559536    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559540    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559543    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559546    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559548    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559551    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559554    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559557    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559562    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559565    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559568    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559573    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559576    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559579    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559582    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559585    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559588    2578 flags.go:64] FLAG: --v="2"
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559592    2578 flags.go:64] FLAG: --version="false"
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559596    2578 flags.go:64] FLAG: --vmodule=""
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559599    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.559602    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 15:20:09.567460 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559716    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559721    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559725    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559728    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559732    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559734    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559737    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559740    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559742    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559745    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559748    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559751    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559755    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559758    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559762    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559765    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559768    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559771    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559774    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 15:20:09.568052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559777    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559780    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559783    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559786    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559789    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559791    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559794    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559796    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559799    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559801    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559804    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559806    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559809    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559812    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559814    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559817    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559819    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559822    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559825    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 15:20:09.568548 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559827    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559830    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559833    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559837    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559840    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559843    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559848    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559850    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559853    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559856    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559858    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559861    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559864    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559866    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559868    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559871    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559874    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559876    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559879    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559881    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 15:20:09.569030 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559884    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559887    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559889    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559892    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559894    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559897    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559899    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559902    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559904    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559907    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559910    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559912    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559915    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559918    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559920    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559923    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559926    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559928    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559933    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559935    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 15:20:09.569555 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559938    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559940    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559943    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559946    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559948    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559951    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559953    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559956    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559958    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559961    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559963    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559966    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559968    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559971    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559973    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559976    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559978    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559981    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559983    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559986    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 15:20:09.570046 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.559989    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 15:20:09.570639 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.560564    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 15:20:09.570639 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.567493    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 15:20:09.570639 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.567509    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 15:20:09.570639 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.567529    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 15:20:09.570639 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.567535    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 15:20:09.570639 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567598    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 15:20:09.570639 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567603    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 15:20:09.570639 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567606    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 15:20:09.570639 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567609    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 15:20:09.570639 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567612    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 15:20:09.570639 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567615    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 15:20:09.570639 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567618    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 15:20:09.570639 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567620    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 15:20:09.570639 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567623    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 15:20:09.570639 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567625    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567628    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567631    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567633    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567636    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567639    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567642    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567646    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567650    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567653    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567656    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567658    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567661    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567664    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567667    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567669    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567672    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567674    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567677    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 15:20:09.571015 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567680    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567686    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567689    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567692    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567694    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567697    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567701    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567705    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567707    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567710    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567713    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567716    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567719    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567721    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567724    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567727    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567729    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567732    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567735    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567738    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 15:20:09.571526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567740    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567743    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567745    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567747    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567750    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567752    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567755    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567758    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567760    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567763    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567765    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567768    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567771    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567774    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567777    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567781    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567785    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567787    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567790    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567793    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 15:20:09.572009 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567795    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567797    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567800    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567803    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567805    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567808    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567810    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567813    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567815    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567818    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567820    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567823    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567825    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567827    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567830    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567833    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567835    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567838    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567841    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567844    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 15:20:09.572508 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567846    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 15:20:09.572998 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567849    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 15:20:09.572998 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567852    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 15:20:09.572998 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567855    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 15:20:09.572998 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567857    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 15:20:09.572998 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567860    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 15:20:09.572998 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567862    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 15:20:09.572998 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567866    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 15:20:09.572998 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567868    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 15:20:09.572998 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567871    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 15:20:09.572998 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.567873    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 15:20:09.572998 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.567879    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 15:20:09.572998 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568009    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 15:20:09.572998 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568014    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 15:20:09.572998 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568017    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 15:20:09.572998 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568020    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568023    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568025    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568028    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568030    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568033    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568035    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568037    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568040    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568042    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568045    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568047    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568050    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568053    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568056    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568060    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568064    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568067    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568070    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 15:20:09.573396 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568072    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568075    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568077    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568080    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568082    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568084    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568088    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568090    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568092    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568095    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568097    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568100    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568102    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568105    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568107    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568109    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568112    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568128    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568131    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568133    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 15:20:09.573871 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568136    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568138    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568141    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568143    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568146    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568148    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568151    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568154    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568157    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568160    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568162    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568165    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568167    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568170    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568172    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568175    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568177    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568180    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568182    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568186    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 15:20:09.574383 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568188    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568191    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568193    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568196    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568198    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568201    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568203    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568205    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568208    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568210    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568212    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568215    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568218    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568221    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568223    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568226    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568230    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568233    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568236    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568238    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 15:20:09.574870 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568241    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 15:20:09.575369 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568244    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 15:20:09.575369 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568246    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 15:20:09.575369 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568249    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 15:20:09.575369 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568252    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 15:20:09.575369 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568255    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 15:20:09.575369 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568257    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 15:20:09.575369 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568260    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 15:20:09.575369 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568263    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 15:20:09.575369 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568266    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 15:20:09.575369 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568270    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 15:20:09.575369 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568272    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 15:20:09.575369 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568276    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 15:20:09.575369 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568278    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 15:20:09.575369 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568281    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 15:20:09.575369 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568284    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 15:20:09.575369 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:09.568287    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 15:20:09.575759 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.568292    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 15:20:09.575759 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.568393    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 15:20:09.575759 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.570299    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 15:20:09.575759 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.571146    2578 server.go:1019] "Starting client certificate rotation"
Sep 09 15:20:09.575759 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.571242    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 15:20:09.575759 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.571775    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 15:20:09.575759 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.573424    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 15:20:09.577897 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.577875    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 15:20:09.593061 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.593040    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 09 15:20:09.595503 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.595485    2578 log.go:25] "Validated CRI v1 image API"
Sep 09 15:20:09.596350 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.596335    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 15:20:09.599727 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.599707    2578 fs.go:135] Filesystem UUIDs: map[3d0af211-a553-47dc-9755-e58b58dd2e9e:/dev/nvme0n1p3 49b863c9-81d6-4077-94a9-921300ca21c7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 09 15:20:09.599803 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.599725    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 15:20:09.602580 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.602561    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 15:20:09.605075 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.604954    2578 manager.go:216] Machine: {Timestamp:2026-09-09 15:20:09.603743217 +0000 UTC m=+0.319645653 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3119109 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20970f470c8d1080e26665eede0c75 SystemUUID:ec20970f-470c-8d10-80e2-6665eede0c75 BootID:de4f6707-cbfc-4351-89d0-753445f3c971 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ae:4e:fe:06:73 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ae:4e:fe:06:73 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:fe:42:0b:66:8b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 15:20:09.605075 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.605068    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 15:20:09.605206 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.605163    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 15:20:09.607902 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.607880    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 15:20:09.608029 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.607904    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-183.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 15:20:09.608550 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.608540    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 15:20:09.608582 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.608553    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 15:20:09.608582 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.608561    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 15:20:09.609408 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.609398    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 15:20:09.610485 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.610476    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 15:20:09.614856 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.614845    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 15:20:09.614973 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.614963    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 15:20:09.615010 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.614991    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 15:20:09.615010 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.615005    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 15:20:09.615064 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.615019    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 15:20:09.615064 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.615029    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 09 15:20:09.615064 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.615038    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 15:20:09.615472 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.615458    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 15:20:09.615519 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.615485    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 15:20:09.618335 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.618154    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 15:20:09.620420 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.620406    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 15:20:09.620473 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.620436    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 15:20:09.620584 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.620575    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 15:20:09.620638 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.620589    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 15:20:09.620638 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.620596    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 15:20:09.620638 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.620601    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 15:20:09.620638 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.620608    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 15:20:09.620638 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.620613    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 15:20:09.621232 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.621221    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 15:20:09.621273 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.621232    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 15:20:09.621273 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.621240    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 15:20:09.621273 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.621245    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 15:20:09.621273 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.621254    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 15:20:09.621273 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.621261    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 15:20:09.622034 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.622019    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 15:20:09.622071 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.622039    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 15:20:09.625901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.625890    2578 server.go:1268] "Started kubelet"
Sep 09 15:20:09.626006 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.625983    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 15:20:09.626077 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.626026    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 15:20:09.626178 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.626158    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 15:20:09.626660 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.626639    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-183.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 15:20:09.626751 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:09.626708    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 15:20:09.626807 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:09.626750    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 15:20:09.626932 ip-10-0-134-183 systemd[1]: Started Kubernetes Kubelet.
Sep 09 15:20:09.627391 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.627247    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 15:20:09.628357 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.628343    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 15:20:09.631818 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:09.631796    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 15:20:09.633246 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.633227    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 15:20:09.633246 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.633234    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 15:20:09.633494 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:09.633433    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
Sep 09 15:20:09.633854 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.633838    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 15:20:09.633854 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.633856    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 15:20:09.633994 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.633872    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 15:20:09.633994 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.633965    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 15:20:09.633994 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.633976    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 15:20:09.634221 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.634205    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 15:20:09.634221 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.634222    2578 factory.go:55] Registering systemd factory
Sep 09 15:20:09.634351 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.634231    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 09 15:20:09.634453 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.634441    2578 factory.go:153] Registering CRI-O factory
Sep 09 15:20:09.634494 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.634457    2578 factory.go:223] Registration of the crio container factory successfully
Sep 09 15:20:09.634494 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.634475    2578 factory.go:103] Registering Raw factory
Sep 09 15:20:09.634494 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.634491    2578 manager.go:1196] Started watching for new ooms in manager
Sep 09 15:20:09.634789 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.634778    2578 manager.go:349] Starting recovery of all containers
Sep 09 15:20:09.634849 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:09.633782    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-183.ec2.internal.18d3b032e492616d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-183.ec2.internal,UID:ip-10-0-134-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-183.ec2.internal,},FirstTimestamp:2026-09-09 15:20:09.625870701 +0000 UTC m=+0.341773139,LastTimestamp:2026-09-09 15:20:09.625870701 +0000 UTC m=+0.341773139,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-183.ec2.internal,}"
Sep 09 15:20:09.641250 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:09.641228    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 15:20:09.641354 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:09.641334    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 15:20:09.645625 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.645611    2578 manager.go:354] Recovery completed
Sep 09 15:20:09.646027 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.646001    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9wj7h"
Sep 09 15:20:09.649727 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.649715    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:20:09.651489 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.651475    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:20:09.651547 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.651504    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:20:09.651547 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.651514    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:20:09.651620 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.651573    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9wj7h"
Sep 09 15:20:09.652065 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.652043    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 15:20:09.652065 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.652057    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 15:20:09.652065 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.652057    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:20:09.652236 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.652072    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 15:20:09.652698 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.652683    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:20:09.652745 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.652709    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:20:09.652745 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.652719    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:20:09.653221 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:09.653144    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-183.ec2.internal.18d3b032e61949ce  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-183.ec2.internal,UID:ip-10-0-134-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-183.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-183.ec2.internal,},FirstTimestamp:2026-09-09 15:20:09.65148923 +0000 UTC m=+0.367391667,LastTimestamp:2026-09-09 15:20:09.65148923 +0000 UTC m=+0.367391667,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-183.ec2.internal,}"
Sep 09 15:20:09.654793 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.654779    2578 policy_none.go:49] "None policy: Start"
Sep 09 15:20:09.654793 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.654795    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 15:20:09.654885 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.654804    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 15:20:09.656776 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.656763    2578 policy_none.go:47] "Start"
Sep 09 15:20:09.693711 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.693694    2578 manager.go:337] "Starting Device Plugin manager"
Sep 09 15:20:09.718066 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:09.693744    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 15:20:09.718066 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.693758    2578 server.go:87] "Starting device plugin registration server"
Sep 09 15:20:09.718066 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.694140    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 15:20:09.718066 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.694154    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 15:20:09.718066 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.694403    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 15:20:09.718066 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.694494    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 15:20:09.718066 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.694503    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 15:20:09.718066 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:09.694936    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 15:20:09.718066 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:09.694989    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-183.ec2.internal\" not found"
Sep 09 15:20:09.785057 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.785027    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 15:20:09.786343 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.786317    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 15:20:09.786446 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.786363    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 15:20:09.786446 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.786387    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 15:20:09.786446 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:09.786423    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 15:20:09.788686 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.788663    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 15:20:09.794530 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.794515    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:20:09.795315 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.795298    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:20:09.795407 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.795333    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:20:09.795407 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.795349    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:20:09.795407 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.795378    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-183.ec2.internal"
Sep 09 15:20:09.804394 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.804332    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-183.ec2.internal"
Sep 09 15:20:09.804394 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:09.804356    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-183.ec2.internal\": node \"ip-10-0-134-183.ec2.internal\" not found"
Sep 09 15:20:09.830099 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:09.830076    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
Sep 09 15:20:09.887250 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.887209    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal"]
Sep 09 15:20:09.887340 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.887292    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:20:09.888703 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.888686    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:20:09.888783 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.888722    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:20:09.888783 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.888736    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:20:09.891149 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.891135    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:20:09.891244 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.891229    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"
Sep 09 15:20:09.891303 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.891267    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:20:09.891850 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.891837    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:20:09.891910 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.891870    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:20:09.891910 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.891880    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:20:09.891910 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.891842    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:20:09.891991 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.891927    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:20:09.891991 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.891937    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:20:09.894071 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.894055    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal"
Sep 09 15:20:09.894189 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.894080    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 15:20:09.894803 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.894783    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 15:20:09.894864 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.894808    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 15:20:09.894864 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.894817    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeHasSufficientPID"
Sep 09 15:20:09.919685 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:09.919665    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-183.ec2.internal\" not found" node="ip-10-0-134-183.ec2.internal"
Sep 09 15:20:09.924109 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:09.924095    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-183.ec2.internal\" not found" node="ip-10-0-134-183.ec2.internal"
Sep 09 15:20:09.931181 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:09.931155    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
Sep 09 15:20:09.935439 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.935421    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d59a6b49d2c887c886a8231821e4f270-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal\" (UID: \"d59a6b49d2c887c886a8231821e4f270\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"
Sep 09 15:20:09.935523 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.935448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d59a6b49d2c887c886a8231821e4f270-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal\" (UID: \"d59a6b49d2c887c886a8231821e4f270\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"
Sep 09 15:20:09.935523 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:09.935466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f5569ed1edc68219f13a87c1afb087c8-config\") pod \"kube-apiserver-proxy-ip-10-0-134-183.ec2.internal\" (UID: \"f5569ed1edc68219f13a87c1afb087c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal"
Sep 09 15:20:10.031284 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:10.031255    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
Sep 09 15:20:10.035560 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.035528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d59a6b49d2c887c886a8231821e4f270-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal\" (UID: \"d59a6b49d2c887c886a8231821e4f270\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"
Sep 09 15:20:10.035673 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.035573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f5569ed1edc68219f13a87c1afb087c8-config\") pod \"kube-apiserver-proxy-ip-10-0-134-183.ec2.internal\" (UID: \"f5569ed1edc68219f13a87c1afb087c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal"
Sep 09 15:20:10.035673 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.035598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d59a6b49d2c887c886a8231821e4f270-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal\" (UID: \"d59a6b49d2c887c886a8231821e4f270\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"
Sep 09 15:20:10.035673 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.035636    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d59a6b49d2c887c886a8231821e4f270-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal\" (UID: \"d59a6b49d2c887c886a8231821e4f270\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"
Sep 09 15:20:10.035673 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.035640    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d59a6b49d2c887c886a8231821e4f270-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal\" (UID: \"d59a6b49d2c887c886a8231821e4f270\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"
Sep 09 15:20:10.035812 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.035638    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f5569ed1edc68219f13a87c1afb087c8-config\") pod \"kube-apiserver-proxy-ip-10-0-134-183.ec2.internal\" (UID: \"f5569ed1edc68219f13a87c1afb087c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal"
Sep 09 15:20:10.132066 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:10.131983    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
Sep 09 15:20:10.222463 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.222435    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"
Sep 09 15:20:10.227392 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.227273    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal"
Sep 09 15:20:10.233022 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:10.233003    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
Sep 09 15:20:10.333493 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:10.333458    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
Sep 09 15:20:10.433968 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:10.433890    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
Sep 09 15:20:10.472521 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.472497    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 15:20:10.534399 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:10.534370    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
Sep 09 15:20:10.570637 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.570610    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 15:20:10.571070 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.570747    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 15:20:10.571070 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.570802    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 15:20:10.628656 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.628631    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 15:20:10.634867 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:10.634846    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
Sep 09 15:20:10.655095 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.655066    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 15:15:10 +0000 UTC" deadline="2028-05-28 02:37:27.806428724 +0000 UTC"
Sep 09 15:20:10.655095 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.655093    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15035h17m17.151338663s"
Sep 09 15:20:10.735529 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:10.735470    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
Sep 09 15:20:10.835927 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:10.835903    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-183.ec2.internal\" not found"
Sep 09 15:20:10.880532 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.880510    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 15:20:10.934363 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.934338    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal"
Sep 09 15:20:10.946036 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.946014    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 15:20:10.949259 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.949245    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal"
Sep 09 15:20:10.954761 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:10.954749    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 15:20:12.531267 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:12.531239    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5569ed1edc68219f13a87c1afb087c8.slice/crio-0abd92611977f1d141f73f962d0c83774eef956c82ffc9b154a69d34d231a957 WatchSource:0}: Error finding container 0abd92611977f1d141f73f962d0c83774eef956c82ffc9b154a69d34d231a957: Status 404 returned error can't find the container with id 0abd92611977f1d141f73f962d0c83774eef956c82ffc9b154a69d34d231a957
Sep 09 15:20:12.531650 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:12.531636    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59a6b49d2c887c886a8231821e4f270.slice/crio-8f0f0663c249e66d60826c0c499fef5f0736bac74efed0323a54e59d3220653d WatchSource:0}: Error finding container 8f0f0663c249e66d60826c0c499fef5f0736bac74efed0323a54e59d3220653d: Status 404 returned error can't find the container with id 8f0f0663c249e66d60826c0c499fef5f0736bac74efed0323a54e59d3220653d
Sep 09 15:20:12.537305 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.537284    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 15:20:12.690105 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.690007    2578 apiserver.go:52] "Watching apiserver"
Sep 09 15:20:12.698335 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.697761    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 15:20:12.699948 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.699920    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal","openshift-multus/multus-fvk5x","openshift-multus/network-metrics-daemon-nhtcj","openshift-network-diagnostics/network-check-target-7x79b","kube-system/global-pull-secret-syncer-h4r6t","kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal","openshift-image-registry/node-ca-qn7pt","openshift-multus/multus-additional-cni-plugins-2wkxv","openshift-network-operator/iptables-alerter-ws5dd","openshift-ovn-kubernetes/ovnkube-node-42r95","kube-system/konnectivity-agent-7kt4h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78","openshift-cluster-node-tuning-operator/tuned-h7ljr","openshift-dns/node-resolver-v2hbm"]
Sep 09 15:20:12.703872 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.703854    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.706075 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.706053    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qn7pt"
Sep 09 15:20:12.706516 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.706495    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 15:20:12.706615 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.706596    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 15:20:12.706681 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.706597    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 15:20:12.706681 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.706617    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 15:20:12.706904 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.706889    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tvrrz\""
Sep 09 15:20:12.707149 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.707134    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 15:20:12.707220 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.707137    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 15:20:12.707683 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.707669    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 15:20:12.707995 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.707964    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 15:20:12.708160 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.708138    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 15:20:12.708381 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.708337    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 15:20:12.708381 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.708368    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hzmfh\""
Sep 09 15:20:12.708381 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.708374    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:12.708558 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:12.708438    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:20:12.710550 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.710533    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.712624 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.712606    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 15:20:12.712708 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.712681    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 15:20:12.712708 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.712689    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 15:20:12.712814 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.712708    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6lh59\""
Sep 09 15:20:12.712814 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.712681    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 15:20:12.712942 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.712927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.713007 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.712993    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 15:20:12.715008 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.714965    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 15:20:12.715118 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.715106    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sh9tj\""
Sep 09 15:20:12.715171 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.715131    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 15:20:12.715281 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.715253    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.715403 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.715383    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 15:20:12.717317 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.717303    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ltvqf\""
Sep 09 15:20:12.717403 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.717390    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 15:20:12.717454 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.717443    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 15:20:12.717702 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.717688    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ws5dd"
Sep 09 15:20:12.717702 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.717697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-var-lib-openvswitch\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.717819 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.717717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-log-socket\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.717819 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.717734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.717819 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.717755    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-env-overrides\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.717935 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.717815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-slash\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.717935 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.717839    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-ovnkube-config\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.717935 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.717856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-ovnkube-script-lib\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.717935 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.717872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b0bf69a-0e36-446f-8748-a84c778fe36e-host\") pod \"node-ca-qn7pt\" (UID: \"0b0bf69a-0e36-446f-8748-a84c778fe36e\") " pod="openshift-image-registry/node-ca-qn7pt"
Sep 09 15:20:12.717935 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.717907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b0bf69a-0e36-446f-8748-a84c778fe36e-serviceca\") pod \"node-ca-qn7pt\" (UID: \"0b0bf69a-0e36-446f-8748-a84c778fe36e\") " pod="openshift-image-registry/node-ca-qn7pt"
Sep 09 15:20:12.717935 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.717935    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-run-ovn\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.718144 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.717951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-node-log\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.718144 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.717969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-cni-bin\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.718144 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.718014    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-cni-netd\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.718144 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.718037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-ovn-node-metrics-cert\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.718144 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.718059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdf4r\" (UniqueName: \"kubernetes.io/projected/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-kube-api-access-gdf4r\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.718144 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.718104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qfdx\" (UniqueName: \"kubernetes.io/projected/0b0bf69a-0e36-446f-8748-a84c778fe36e-kube-api-access-8qfdx\") pod \"node-ca-qn7pt\" (UID: \"0b0bf69a-0e36-446f-8748-a84c778fe36e\") " pod="openshift-image-registry/node-ca-qn7pt"
Sep 09 15:20:12.718144 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.718141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs\") pod \"network-metrics-daemon-nhtcj\" (UID: \"7052f7a0-16d5-4115-82ae-34f07e9db9f9\") " pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:12.718394 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.718166    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-kubelet\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.718394 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.718191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-systemd-units\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.718394 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.718241    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-etc-openvswitch\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.718394 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.718261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-run-openvswitch\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.718394 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.718277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.718394 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.718311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4x9g\" (UniqueName: \"kubernetes.io/projected/7052f7a0-16d5-4115-82ae-34f07e9db9f9-kube-api-access-n4x9g\") pod \"network-metrics-daemon-nhtcj\" (UID: \"7052f7a0-16d5-4115-82ae-34f07e9db9f9\") " pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:12.718394 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.718335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-run-netns\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.718394 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.718367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-run-systemd\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.718625 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.718520    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 15:20:12.719612 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.719595    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 15:20:12.719657 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.719615    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t2bzg\""
Sep 09 15:20:12.719779 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.719764    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 15:20:12.719937 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.719921    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.719937 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.719930    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 15:20:12.721787 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.721772    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-czbt4\""
Sep 09 15:20:12.722080 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.722056    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 15:20:12.722270 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.722252    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7kt4h"
Sep 09 15:20:12.724694 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.724674    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8n4kk\""
Sep 09 15:20:12.724786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.724718    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:12.724786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.724763    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 15:20:12.724944 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:12.724770    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4r6t" podUID="560288f5-fbb4-4d6b-84bd-0987afe4c482"
Sep 09 15:20:12.724944 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.724838    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 15:20:12.727519 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.727500    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:12.727608 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:12.727565    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7x79b" podUID="b1406d46-d991-410d-bc6d-be16046a2fd5"
Sep 09 15:20:12.729713 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.729698    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v2hbm"
Sep 09 15:20:12.731755 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.731738    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 15:20:12.731954 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.731939    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nvh5t\""
Sep 09 15:20:12.732338 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.732310    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 15:20:12.808924 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.808900    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 15:20:12.819522 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv4l6\" (UniqueName: \"kubernetes.io/projected/adc5a00c-c7b6-42c7-9f72-1c86bcb50e38-kube-api-access-qv4l6\") pod \"iptables-alerter-ws5dd\" (UID: \"adc5a00c-c7b6-42c7-9f72-1c86bcb50e38\") " pod="openshift-network-operator/iptables-alerter-ws5dd"
Sep 09 15:20:12.819632 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-multus-cni-dir\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.819632 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-etc-kubernetes\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.819632 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc01e4b-4a6f-47f0-8220-ca62e74d7250-tmp-dir\") pod \"node-resolver-v2hbm\" (UID: \"9fc01e4b-4a6f-47f0-8220-ca62e74d7250\") " pod="openshift-dns/node-resolver-v2hbm"
Sep 09 15:20:12.819632 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdf4r\" (UniqueName: \"kubernetes.io/projected/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-kube-api-access-gdf4r\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.819632 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-etc-openvswitch\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.819880 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44d88cca-0066-4eb8-8cb2-39c71959ec98-device-dir\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.819880 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-host-var-lib-cni-multus\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.819880 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-run-systemd\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.819880 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b0bf69a-0e36-446f-8748-a84c778fe36e-serviceca\") pod \"node-ca-qn7pt\" (UID: \"0b0bf69a-0e36-446f-8748-a84c778fe36e\") " pod="openshift-image-registry/node-ca-qn7pt"
Sep 09 15:20:12.819880 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819709    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-etc-openvswitch\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.819880 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819719    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/879098a0-7a7e-4b71-b0ee-b7c759162e26-system-cni-dir\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.819880 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819753    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-run-systemd\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.819880 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/879098a0-7a7e-4b71-b0ee-b7c759162e26-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.819880 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-kubernetes\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.819880 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-systemd\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.820405 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-run\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.820405 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-cni-binary-copy\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.820405 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/879098a0-7a7e-4b71-b0ee-b7c759162e26-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.820405 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.819963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44d88cca-0066-4eb8-8cb2-39c71959ec98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.820405 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-multus-conf-dir\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.820405 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/588589aa-effd-4129-96af-6180f4a1b5d8-konnectivity-ca\") pod \"konnectivity-agent-7kt4h\" (UID: \"588589aa-effd-4129-96af-6180f4a1b5d8\") " pod="kube-system/konnectivity-agent-7kt4h"
Sep 09 15:20:12.820405 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl27s\" (UniqueName: \"kubernetes.io/projected/9fc01e4b-4a6f-47f0-8220-ca62e74d7250-kube-api-access-dl27s\") pod \"node-resolver-v2hbm\" (UID: \"9fc01e4b-4a6f-47f0-8220-ca62e74d7250\") " pod="openshift-dns/node-resolver-v2hbm"
Sep 09 15:20:12.820405 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-cni-bin\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.820405 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qfdx\" (UniqueName: \"kubernetes.io/projected/0b0bf69a-0e36-446f-8748-a84c778fe36e-kube-api-access-8qfdx\") pod \"node-ca-qn7pt\" (UID: \"0b0bf69a-0e36-446f-8748-a84c778fe36e\") " pod="openshift-image-registry/node-ca-qn7pt"
Sep 09 15:20:12.820405 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adc5a00c-c7b6-42c7-9f72-1c86bcb50e38-iptables-alerter-script\") pod \"iptables-alerter-ws5dd\" (UID: \"adc5a00c-c7b6-42c7-9f72-1c86bcb50e38\") " pod="openshift-network-operator/iptables-alerter-ws5dd"
Sep 09 15:20:12.820405 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-modprobe-d\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.820405 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820175    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-cni-bin\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.820405 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-host-var-lib-cni-bin\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.820405 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820223    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-host-run-multus-certs\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.820405 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret\") pod \"global-pull-secret-syncer-h4r6t\" (UID: \"560288f5-fbb4-4d6b-84bd-0987afe4c482\") " pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:12.820405 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44d88cca-0066-4eb8-8cb2-39c71959ec98-etc-selinux\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.821125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820323    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-sys\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.821125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-host-run-k8s-cni-cncf-io\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.821125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/588589aa-effd-4129-96af-6180f4a1b5d8-agent-certs\") pod \"konnectivity-agent-7kt4h\" (UID: \"588589aa-effd-4129-96af-6180f4a1b5d8\") " pod="kube-system/konnectivity-agent-7kt4h"
Sep 09 15:20:12.821125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-var-lib-openvswitch\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.821125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/879098a0-7a7e-4b71-b0ee-b7c759162e26-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.821125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-sysconfig\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.821125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-host-var-lib-kubelet\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.821125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-slash\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.821125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820520    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-var-lib-openvswitch\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.821125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820526    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-ovnkube-config\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.821125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-sysctl-conf\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.821125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-slash\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.821125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-run-ovn\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.821125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-node-log\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.821125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-cni-netd\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.821125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/879098a0-7a7e-4b71-b0ee-b7c759162e26-cni-binary-copy\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.821125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820646    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b0bf69a-0e36-446f-8748-a84c778fe36e-serviceca\") pod \"node-ca-qn7pt\" (UID: \"0b0bf69a-0e36-446f-8748-a84c778fe36e\") " pod="openshift-image-registry/node-ca-qn7pt"
Sep 09 15:20:12.821855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-run-ovn\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.821855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnks7\" (UniqueName: \"kubernetes.io/projected/44d88cca-0066-4eb8-8cb2-39c71959ec98-kube-api-access-gnks7\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.821855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-node-log\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.821855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820691    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-cni-netd\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.821855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-multus-socket-dir-parent\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.821855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/560288f5-fbb4-4d6b-84bd-0987afe4c482-kubelet-config\") pod \"global-pull-secret-syncer-h4r6t\" (UID: \"560288f5-fbb4-4d6b-84bd-0987afe4c482\") " pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:12.821855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-lib-modules\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.821855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820797    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-systemd-units\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.821855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820812    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhkhs\" (UniqueName: \"kubernetes.io/projected/879098a0-7a7e-4b71-b0ee-b7c759162e26-kube-api-access-zhkhs\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.821855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820830    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44d88cca-0066-4eb8-8cb2-39c71959ec98-sys-fs\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.821855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-sysctl-d\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.821855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820847    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-systemd-units\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.821855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sjhb\" (UniqueName: \"kubernetes.io/projected/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-kube-api-access-9sjhb\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.821855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820877    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-cnibin\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.821855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820909    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-run-netns\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.821855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-ovnkube-script-lib\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.821855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fwbg\" (UniqueName: \"kubernetes.io/projected/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-kube-api-access-2fwbg\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.822432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.820949    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-run-netns\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.822432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nms7\" (UniqueName: \"kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7\") pod \"network-check-target-7x79b\" (UID: \"b1406d46-d991-410d-bc6d-be16046a2fd5\") " pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:12.822432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b0bf69a-0e36-446f-8748-a84c778fe36e-host\") pod \"node-ca-qn7pt\" (UID: \"0b0bf69a-0e36-446f-8748-a84c778fe36e\") " pod="openshift-image-registry/node-ca-qn7pt"
Sep 09 15:20:12.822432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-ovnkube-config\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.822432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-var-lib-kubelet\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.822432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-host-run-netns\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.822432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821171    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b0bf69a-0e36-446f-8748-a84c778fe36e-host\") pod \"node-ca-qn7pt\" (UID: \"0b0bf69a-0e36-446f-8748-a84c778fe36e\") " pod="openshift-image-registry/node-ca-qn7pt"
Sep 09 15:20:12.822432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fc01e4b-4a6f-47f0-8220-ca62e74d7250-hosts-file\") pod \"node-resolver-v2hbm\" (UID: \"9fc01e4b-4a6f-47f0-8220-ca62e74d7250\") " pod="openshift-dns/node-resolver-v2hbm"
Sep 09 15:20:12.822432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-host\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.822432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/879098a0-7a7e-4b71-b0ee-b7c759162e26-cnibin\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.822432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44d88cca-0066-4eb8-8cb2-39c71959ec98-socket-dir\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.822432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44d88cca-0066-4eb8-8cb2-39c71959ec98-registration-dir\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.822432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821336    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-tmp\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.822432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-ovn-node-metrics-cert\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.822432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs\") pod \"network-metrics-daemon-nhtcj\" (UID: \"7052f7a0-16d5-4115-82ae-34f07e9db9f9\") " pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:12.822432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821469    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-ovnkube-script-lib\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.822432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-kubelet\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.822901 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:12.821590    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:12.822901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821614    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-kubelet\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.822901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-run-openvswitch\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.822901 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:12.821664    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs podName:7052f7a0-16d5-4115-82ae-34f07e9db9f9 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:13.321644669 +0000 UTC m=+2.963686636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs") pod "network-metrics-daemon-nhtcj" (UID: "7052f7a0-16d5-4115-82ae-34f07e9db9f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:12.822901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.822901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-run-openvswitch\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.822901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adc5a00c-c7b6-42c7-9f72-1c86bcb50e38-host-slash\") pod \"iptables-alerter-ws5dd\" (UID: \"adc5a00c-c7b6-42c7-9f72-1c86bcb50e38\") " pod="openshift-network-operator/iptables-alerter-ws5dd"
Sep 09 15:20:12.822901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-tuned\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.822901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821746    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.822901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-system-cni-dir\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.822901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821797    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4x9g\" (UniqueName: \"kubernetes.io/projected/7052f7a0-16d5-4115-82ae-34f07e9db9f9-kube-api-access-n4x9g\") pod \"network-metrics-daemon-nhtcj\" (UID: \"7052f7a0-16d5-4115-82ae-34f07e9db9f9\") " pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:12.822901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-log-socket\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.822901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.822901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-log-socket\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.822901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-env-overrides\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.822901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-hostroot\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.822901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.823428 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-multus-daemon-config\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.823428 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/560288f5-fbb4-4d6b-84bd-0987afe4c482-dbus\") pod \"global-pull-secret-syncer-h4r6t\" (UID: \"560288f5-fbb4-4d6b-84bd-0987afe4c482\") " pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:12.823428 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.821999    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 15:20:12.823428 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.822186    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-env-overrides\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.825697 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.825513    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-ovn-node-metrics-cert\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.827351 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.827332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdf4r\" (UniqueName: \"kubernetes.io/projected/8e8d5b7a-7aa0-40dd-951e-76e34bf395cb-kube-api-access-gdf4r\") pod \"ovnkube-node-42r95\" (UID: \"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:12.827519 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.827497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qfdx\" (UniqueName: \"kubernetes.io/projected/0b0bf69a-0e36-446f-8748-a84c778fe36e-kube-api-access-8qfdx\") pod \"node-ca-qn7pt\" (UID: \"0b0bf69a-0e36-446f-8748-a84c778fe36e\") " pod="openshift-image-registry/node-ca-qn7pt"
Sep 09 15:20:12.828159 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.828144    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t2jcs"
Sep 09 15:20:12.831635 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.831616    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4x9g\" (UniqueName: \"kubernetes.io/projected/7052f7a0-16d5-4115-82ae-34f07e9db9f9-kube-api-access-n4x9g\") pod \"network-metrics-daemon-nhtcj\" (UID: \"7052f7a0-16d5-4115-82ae-34f07e9db9f9\") " pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:12.835184 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.835165    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t2jcs"
Sep 09 15:20:12.866925 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.866886    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal" event={"ID":"d59a6b49d2c887c886a8231821e4f270","Type":"ContainerStarted","Data":"8f0f0663c249e66d60826c0c499fef5f0736bac74efed0323a54e59d3220653d"}
Sep 09 15:20:12.868046 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.868026    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal" event={"ID":"f5569ed1edc68219f13a87c1afb087c8","Type":"ContainerStarted","Data":"0abd92611977f1d141f73f962d0c83774eef956c82ffc9b154a69d34d231a957"}
Sep 09 15:20:12.922734 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.922709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-sysctl-conf\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.922843 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.922738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/879098a0-7a7e-4b71-b0ee-b7c759162e26-cni-binary-copy\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.922843 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.922754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnks7\" (UniqueName: \"kubernetes.io/projected/44d88cca-0066-4eb8-8cb2-39c71959ec98-kube-api-access-gnks7\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.922843 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.922769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-multus-socket-dir-parent\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.922843 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.922789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/560288f5-fbb4-4d6b-84bd-0987afe4c482-kubelet-config\") pod \"global-pull-secret-syncer-h4r6t\" (UID: \"560288f5-fbb4-4d6b-84bd-0987afe4c482\") " pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:12.922843 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.922805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-lib-modules\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.922843 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.922821    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhkhs\" (UniqueName: \"kubernetes.io/projected/879098a0-7a7e-4b71-b0ee-b7c759162e26-kube-api-access-zhkhs\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.922843 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.922837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44d88cca-0066-4eb8-8cb2-39c71959ec98-sys-fs\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.923247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.922857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-sysctl-d\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.923247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.922879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sjhb\" (UniqueName: \"kubernetes.io/projected/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-kube-api-access-9sjhb\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.923247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.922902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-cnibin\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.923247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.922913    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/560288f5-fbb4-4d6b-84bd-0987afe4c482-kubelet-config\") pod \"global-pull-secret-syncer-h4r6t\" (UID: \"560288f5-fbb4-4d6b-84bd-0987afe4c482\") " pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:12.923247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.922926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fwbg\" (UniqueName: \"kubernetes.io/projected/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-kube-api-access-2fwbg\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.923247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.922933    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-multus-socket-dir-parent\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.923247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.922949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nms7\" (UniqueName: \"kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7\") pod \"network-check-target-7x79b\" (UID: \"b1406d46-d991-410d-bc6d-be16046a2fd5\") " pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:12.923247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.922949    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-lib-modules\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.923247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.922998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44d88cca-0066-4eb8-8cb2-39c71959ec98-sys-fs\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.923247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.922922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-sysctl-conf\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.923247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-cnibin\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.923247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-var-lib-kubelet\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.923247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-host-run-netns\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.923247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fc01e4b-4a6f-47f0-8220-ca62e74d7250-hosts-file\") pod \"node-resolver-v2hbm\" (UID: \"9fc01e4b-4a6f-47f0-8220-ca62e74d7250\") " pod="openshift-dns/node-resolver-v2hbm"
Sep 09 15:20:12.923247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923045    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-sysctl-d\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.923247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-var-lib-kubelet\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.923247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-host\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.923247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923150    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-host-run-netns\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.924120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/879098a0-7a7e-4b71-b0ee-b7c759162e26-cnibin\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.924120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44d88cca-0066-4eb8-8cb2-39c71959ec98-socket-dir\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.924120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923191    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fc01e4b-4a6f-47f0-8220-ca62e74d7250-hosts-file\") pod \"node-resolver-v2hbm\" (UID: \"9fc01e4b-4a6f-47f0-8220-ca62e74d7250\") " pod="openshift-dns/node-resolver-v2hbm"
Sep 09 15:20:12.924120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44d88cca-0066-4eb8-8cb2-39c71959ec98-registration-dir\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.924120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923221    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-host\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.924120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-tmp\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.924120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923229    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/879098a0-7a7e-4b71-b0ee-b7c759162e26-cnibin\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.924120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923265    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44d88cca-0066-4eb8-8cb2-39c71959ec98-registration-dir\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.924120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923283    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adc5a00c-c7b6-42c7-9f72-1c86bcb50e38-host-slash\") pod \"iptables-alerter-ws5dd\" (UID: \"adc5a00c-c7b6-42c7-9f72-1c86bcb50e38\") " pod="openshift-network-operator/iptables-alerter-ws5dd"
Sep 09 15:20:12.924120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-tuned\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.924120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44d88cca-0066-4eb8-8cb2-39c71959ec98-socket-dir\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.924120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923331    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adc5a00c-c7b6-42c7-9f72-1c86bcb50e38-host-slash\") pod \"iptables-alerter-ws5dd\" (UID: \"adc5a00c-c7b6-42c7-9f72-1c86bcb50e38\") " pod="openshift-network-operator/iptables-alerter-ws5dd"
Sep 09 15:20:12.924120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923339    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-system-cni-dir\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.924120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-hostroot\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.924120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-system-cni-dir\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.924120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-multus-daemon-config\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.924120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923443    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-hostroot\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.924120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/560288f5-fbb4-4d6b-84bd-0987afe4c482-dbus\") pod \"global-pull-secret-syncer-h4r6t\" (UID: \"560288f5-fbb4-4d6b-84bd-0987afe4c482\") " pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:12.925087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv4l6\" (UniqueName: \"kubernetes.io/projected/adc5a00c-c7b6-42c7-9f72-1c86bcb50e38-kube-api-access-qv4l6\") pod \"iptables-alerter-ws5dd\" (UID: \"adc5a00c-c7b6-42c7-9f72-1c86bcb50e38\") " pod="openshift-network-operator/iptables-alerter-ws5dd"
Sep 09 15:20:12.925087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923487    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/560288f5-fbb4-4d6b-84bd-0987afe4c482-dbus\") pod \"global-pull-secret-syncer-h4r6t\" (UID: \"560288f5-fbb4-4d6b-84bd-0987afe4c482\") " pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:12.925087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-multus-cni-dir\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.925087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923511    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/879098a0-7a7e-4b71-b0ee-b7c759162e26-cni-binary-copy\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.925087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-etc-kubernetes\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.925087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc01e4b-4a6f-47f0-8220-ca62e74d7250-tmp-dir\") pod \"node-resolver-v2hbm\" (UID: \"9fc01e4b-4a6f-47f0-8220-ca62e74d7250\") " pod="openshift-dns/node-resolver-v2hbm"
Sep 09 15:20:12.925087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923597    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-multus-cni-dir\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.925087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44d88cca-0066-4eb8-8cb2-39c71959ec98-device-dir\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.925087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-etc-kubernetes\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.925087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-host-var-lib-cni-multus\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.925087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923689    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-host-var-lib-cni-multus\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.925087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44d88cca-0066-4eb8-8cb2-39c71959ec98-device-dir\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.925087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923705    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/879098a0-7a7e-4b71-b0ee-b7c759162e26-system-cni-dir\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.925087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923745    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/879098a0-7a7e-4b71-b0ee-b7c759162e26-system-cni-dir\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.925087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/879098a0-7a7e-4b71-b0ee-b7c759162e26-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.925087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-kubernetes\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.925087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-systemd\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.926073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc01e4b-4a6f-47f0-8220-ca62e74d7250-tmp-dir\") pod \"node-resolver-v2hbm\" (UID: \"9fc01e4b-4a6f-47f0-8220-ca62e74d7250\") " pod="openshift-dns/node-resolver-v2hbm"
Sep 09 15:20:12.926073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-systemd\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.926073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923851    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-kubernetes\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.926073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923882    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-run\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.926073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-cni-binary-copy\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.926073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923933    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/879098a0-7a7e-4b71-b0ee-b7c759162e26-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.926073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923937    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-multus-daemon-config\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.926073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44d88cca-0066-4eb8-8cb2-39c71959ec98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.926073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.923971    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-run\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.926073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-multus-conf-dir\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.926073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/588589aa-effd-4129-96af-6180f4a1b5d8-konnectivity-ca\") pod \"konnectivity-agent-7kt4h\" (UID: \"588589aa-effd-4129-96af-6180f4a1b5d8\") " pod="kube-system/konnectivity-agent-7kt4h"
Sep 09 15:20:12.926073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924038    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44d88cca-0066-4eb8-8cb2-39c71959ec98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.926073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl27s\" (UniqueName: \"kubernetes.io/projected/9fc01e4b-4a6f-47f0-8220-ca62e74d7250-kube-api-access-dl27s\") pod \"node-resolver-v2hbm\" (UID: \"9fc01e4b-4a6f-47f0-8220-ca62e74d7250\") " pod="openshift-dns/node-resolver-v2hbm"
Sep 09 15:20:12.926073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adc5a00c-c7b6-42c7-9f72-1c86bcb50e38-iptables-alerter-script\") pod \"iptables-alerter-ws5dd\" (UID: \"adc5a00c-c7b6-42c7-9f72-1c86bcb50e38\") " pod="openshift-network-operator/iptables-alerter-ws5dd"
Sep 09 15:20:12.926073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-modprobe-d\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.926073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-host-var-lib-cni-bin\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.926073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-host-run-multus-certs\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.926857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret\") pod \"global-pull-secret-syncer-h4r6t\" (UID: \"560288f5-fbb4-4d6b-84bd-0987afe4c482\") " pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:12.926857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924204    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-multus-conf-dir\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.926857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44d88cca-0066-4eb8-8cb2-39c71959ec98-etc-selinux\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.926857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-sys\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.926857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-host-run-k8s-cni-cncf-io\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.926857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/879098a0-7a7e-4b71-b0ee-b7c759162e26-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.926857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/588589aa-effd-4129-96af-6180f4a1b5d8-agent-certs\") pod \"konnectivity-agent-7kt4h\" (UID: \"588589aa-effd-4129-96af-6180f4a1b5d8\") " pod="kube-system/konnectivity-agent-7kt4h"
Sep 09 15:20:12.926857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924297    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-modprobe-d\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.926857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924306    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/879098a0-7a7e-4b71-b0ee-b7c759162e26-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.926857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-sysconfig\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.926857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-host-var-lib-kubelet\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.926857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924410    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-host-var-lib-kubelet\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.926857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-cni-binary-copy\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.926857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-host-var-lib-cni-bin\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.926857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924483    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-host-run-multus-certs\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.926857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924514    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-host-run-k8s-cni-cncf-io\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.926857 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:12.924528    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:12.926857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924567    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44d88cca-0066-4eb8-8cb2-39c71959ec98-etc-selinux\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.927371 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:12.924591    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret podName:560288f5-fbb4-4d6b-84bd-0987afe4c482 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:13.42457335 +0000 UTC m=+3.066615330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret") pod "global-pull-secret-syncer-h4r6t" (UID: "560288f5-fbb4-4d6b-84bd-0987afe4c482") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:12.927371 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/588589aa-effd-4129-96af-6180f4a1b5d8-konnectivity-ca\") pod \"konnectivity-agent-7kt4h\" (UID: \"588589aa-effd-4129-96af-6180f4a1b5d8\") " pod="kube-system/konnectivity-agent-7kt4h"
Sep 09 15:20:12.927371 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-sys\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.927371 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924664    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/879098a0-7a7e-4b71-b0ee-b7c759162e26-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.927371 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924714    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/879098a0-7a7e-4b71-b0ee-b7c759162e26-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.927371 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-sysconfig\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.927371 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.924857    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adc5a00c-c7b6-42c7-9f72-1c86bcb50e38-iptables-alerter-script\") pod \"iptables-alerter-ws5dd\" (UID: \"adc5a00c-c7b6-42c7-9f72-1c86bcb50e38\") " pod="openshift-network-operator/iptables-alerter-ws5dd"
Sep 09 15:20:12.927371 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.925731    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-tmp\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.927371 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.926342    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-etc-tuned\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.927371 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.927094    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/588589aa-effd-4129-96af-6180f4a1b5d8-agent-certs\") pod \"konnectivity-agent-7kt4h\" (UID: \"588589aa-effd-4129-96af-6180f4a1b5d8\") " pod="kube-system/konnectivity-agent-7kt4h"
Sep 09 15:20:12.929300 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:12.929281    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:20:12.929394 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:12.929305    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:20:12.929394 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:12.929317    2578 projected.go:196] Error preparing data for projected volume kube-api-access-6nms7 for pod openshift-network-diagnostics/network-check-target-7x79b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:12.929394 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:12.929362    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7 podName:b1406d46-d991-410d-bc6d-be16046a2fd5 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:13.429349577 +0000 UTC m=+3.071391545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6nms7" (UniqueName: "kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7") pod "network-check-target-7x79b" (UID: "b1406d46-d991-410d-bc6d-be16046a2fd5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:12.931299 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.931267    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv4l6\" (UniqueName: \"kubernetes.io/projected/adc5a00c-c7b6-42c7-9f72-1c86bcb50e38-kube-api-access-qv4l6\") pod \"iptables-alerter-ws5dd\" (UID: \"adc5a00c-c7b6-42c7-9f72-1c86bcb50e38\") " pod="openshift-network-operator/iptables-alerter-ws5dd"
Sep 09 15:20:12.931762 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.931741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sjhb\" (UniqueName: \"kubernetes.io/projected/fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7-kube-api-access-9sjhb\") pod \"tuned-h7ljr\" (UID: \"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7\") " pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:12.931845 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.931781    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fwbg\" (UniqueName: \"kubernetes.io/projected/bd6109fe-7e99-4d98-87d0-8a6fc95d6938-kube-api-access-2fwbg\") pod \"multus-fvk5x\" (UID: \"bd6109fe-7e99-4d98-87d0-8a6fc95d6938\") " pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:12.932289 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.932268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnks7\" (UniqueName: \"kubernetes.io/projected/44d88cca-0066-4eb8-8cb2-39c71959ec98-kube-api-access-gnks7\") pod \"aws-ebs-csi-driver-node-qzd78\" (UID: \"44d88cca-0066-4eb8-8cb2-39c71959ec98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:12.932365 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.932331    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhkhs\" (UniqueName: \"kubernetes.io/projected/879098a0-7a7e-4b71-b0ee-b7c759162e26-kube-api-access-zhkhs\") pod \"multus-additional-cni-plugins-2wkxv\" (UID: \"879098a0-7a7e-4b71-b0ee-b7c759162e26\") " pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:12.932415 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.932373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl27s\" (UniqueName: \"kubernetes.io/projected/9fc01e4b-4a6f-47f0-8220-ca62e74d7250-kube-api-access-dl27s\") pod \"node-resolver-v2hbm\" (UID: \"9fc01e4b-4a6f-47f0-8220-ca62e74d7250\") " pod="openshift-dns/node-resolver-v2hbm"
Sep 09 15:20:12.954244 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:12.954190    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 15:20:13.014191 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.014165    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:13.020389 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.020373    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qn7pt"
Sep 09 15:20:13.025121 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:13.025101    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8d5b7a_7aa0_40dd_951e_76e34bf395cb.slice/crio-090bdf9199d630684bb1be375a374faabae1f585502d5c25ac681eaad7c962f8 WatchSource:0}: Error finding container 090bdf9199d630684bb1be375a374faabae1f585502d5c25ac681eaad7c962f8: Status 404 returned error can't find the container with id 090bdf9199d630684bb1be375a374faabae1f585502d5c25ac681eaad7c962f8
Sep 09 15:20:13.026738 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.026662    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2wkxv"
Sep 09 15:20:13.028957 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:13.028934    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0bf69a_0e36_446f_8748_a84c778fe36e.slice/crio-09808586516e20dc7c600c97c3d918da681edc3a563f39a9c9e3f9f04e99c6b7 WatchSource:0}: Error finding container 09808586516e20dc7c600c97c3d918da681edc3a563f39a9c9e3f9f04e99c6b7: Status 404 returned error can't find the container with id 09808586516e20dc7c600c97c3d918da681edc3a563f39a9c9e3f9f04e99c6b7
Sep 09 15:20:13.031524 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.031508    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78"
Sep 09 15:20:13.035052 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:13.035035    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879098a0_7a7e_4b71_b0ee_b7c759162e26.slice/crio-fe1feda9dc864c2f89c862e82f5a1cbd1b757090b451e56112fa33a54348180b WatchSource:0}: Error finding container fe1feda9dc864c2f89c862e82f5a1cbd1b757090b451e56112fa33a54348180b: Status 404 returned error can't find the container with id fe1feda9dc864c2f89c862e82f5a1cbd1b757090b451e56112fa33a54348180b
Sep 09 15:20:13.040119 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.040101    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h7ljr"
Sep 09 15:20:13.040924 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:13.040902    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d88cca_0066_4eb8_8cb2_39c71959ec98.slice/crio-81e4b791c44be43fe9e1a136a06feaee44269364198903a4250121604b3792d7 WatchSource:0}: Error finding container 81e4b791c44be43fe9e1a136a06feaee44269364198903a4250121604b3792d7: Status 404 returned error can't find the container with id 81e4b791c44be43fe9e1a136a06feaee44269364198903a4250121604b3792d7
Sep 09 15:20:13.045652 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.045635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ws5dd"
Sep 09 15:20:13.048600 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:13.048585    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5bf48e_69b5_4bfc_8a0b_4d9d6a8a6bc7.slice/crio-a964f6d795aad217af49897fab9585c5c6c62b43cd531684c18fe34cd8388709 WatchSource:0}: Error finding container a964f6d795aad217af49897fab9585c5c6c62b43cd531684c18fe34cd8388709: Status 404 returned error can't find the container with id a964f6d795aad217af49897fab9585c5c6c62b43cd531684c18fe34cd8388709
Sep 09 15:20:13.050544 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.050489    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fvk5x"
Sep 09 15:20:13.054410 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:13.054391    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc5a00c_c7b6_42c7_9f72_1c86bcb50e38.slice/crio-023be9c749758f9324d2df82f3c50183d928ba4b28620fe5b8562f3389e20819 WatchSource:0}: Error finding container 023be9c749758f9324d2df82f3c50183d928ba4b28620fe5b8562f3389e20819: Status 404 returned error can't find the container with id 023be9c749758f9324d2df82f3c50183d928ba4b28620fe5b8562f3389e20819
Sep 09 15:20:13.055633 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.055615    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7kt4h"
Sep 09 15:20:13.059087 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:13.059067    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6109fe_7e99_4d98_87d0_8a6fc95d6938.slice/crio-dbb3213e251f78b2fda231620d0b8e6cc729cde0dae1699bd256bb42806c0930 WatchSource:0}: Error finding container dbb3213e251f78b2fda231620d0b8e6cc729cde0dae1699bd256bb42806c0930: Status 404 returned error can't find the container with id dbb3213e251f78b2fda231620d0b8e6cc729cde0dae1699bd256bb42806c0930
Sep 09 15:20:13.059920 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.059906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v2hbm"
Sep 09 15:20:13.066050 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:13.066033    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588589aa_effd_4129_96af_6180f4a1b5d8.slice/crio-66616b0404fe08c7cb4fdc5855ac394f01ae1b14b02ceed44c546d89f5586b41 WatchSource:0}: Error finding container 66616b0404fe08c7cb4fdc5855ac394f01ae1b14b02ceed44c546d89f5586b41: Status 404 returned error can't find the container with id 66616b0404fe08c7cb4fdc5855ac394f01ae1b14b02ceed44c546d89f5586b41
Sep 09 15:20:13.068942 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:13.068923    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc01e4b_4a6f_47f0_8220_ca62e74d7250.slice/crio-16d4b365b202111d961312947519ab190d0364baba4a6727369771691e73630f WatchSource:0}: Error finding container 16d4b365b202111d961312947519ab190d0364baba4a6727369771691e73630f: Status 404 returned error can't find the container with id 16d4b365b202111d961312947519ab190d0364baba4a6727369771691e73630f
Sep 09 15:20:13.326833 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.326745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs\") pod \"network-metrics-daemon-nhtcj\" (UID: \"7052f7a0-16d5-4115-82ae-34f07e9db9f9\") " pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:13.326974 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:13.326919    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:13.327030 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:13.327013    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs podName:7052f7a0-16d5-4115-82ae-34f07e9db9f9 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:14.326993256 +0000 UTC m=+3.969035238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs") pod "network-metrics-daemon-nhtcj" (UID: "7052f7a0-16d5-4115-82ae-34f07e9db9f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:13.428857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.427849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret\") pod \"global-pull-secret-syncer-h4r6t\" (UID: \"560288f5-fbb4-4d6b-84bd-0987afe4c482\") " pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:13.428857 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:13.428055    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:13.428857 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:13.428122    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret podName:560288f5-fbb4-4d6b-84bd-0987afe4c482 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:14.428104988 +0000 UTC m=+4.070146973 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret") pod "global-pull-secret-syncer-h4r6t" (UID: "560288f5-fbb4-4d6b-84bd-0987afe4c482") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:13.529076 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.529047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nms7\" (UniqueName: \"kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7\") pod \"network-check-target-7x79b\" (UID: \"b1406d46-d991-410d-bc6d-be16046a2fd5\") " pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:13.529258 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:13.529202    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:20:13.529258 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:13.529228    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:20:13.529258 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:13.529239    2578 projected.go:196] Error preparing data for projected volume kube-api-access-6nms7 for pod openshift-network-diagnostics/network-check-target-7x79b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:13.529409 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:13.529291    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7 podName:b1406d46-d991-410d-bc6d-be16046a2fd5 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:14.529276998 +0000 UTC m=+4.171318978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6nms7" (UniqueName: "kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7") pod "network-check-target-7x79b" (UID: "b1406d46-d991-410d-bc6d-be16046a2fd5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:13.839432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.839388    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 15:15:12 +0000 UTC" deadline="2028-03-03 11:54:13.347459823 +0000 UTC"
Sep 09 15:20:13.839432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.839432    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12980h33m59.508032906s"
Sep 09 15:20:13.875805 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.875765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v2hbm" event={"ID":"9fc01e4b-4a6f-47f0-8220-ca62e74d7250","Type":"ContainerStarted","Data":"16d4b365b202111d961312947519ab190d0364baba4a6727369771691e73630f"}
Sep 09 15:20:13.879349 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.879318    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7kt4h" event={"ID":"588589aa-effd-4129-96af-6180f4a1b5d8","Type":"ContainerStarted","Data":"66616b0404fe08c7cb4fdc5855ac394f01ae1b14b02ceed44c546d89f5586b41"}
Sep 09 15:20:13.882309 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.882286    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fvk5x" event={"ID":"bd6109fe-7e99-4d98-87d0-8a6fc95d6938","Type":"ContainerStarted","Data":"dbb3213e251f78b2fda231620d0b8e6cc729cde0dae1699bd256bb42806c0930"}
Sep 09 15:20:13.883671 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.883650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ws5dd" event={"ID":"adc5a00c-c7b6-42c7-9f72-1c86bcb50e38","Type":"ContainerStarted","Data":"023be9c749758f9324d2df82f3c50183d928ba4b28620fe5b8562f3389e20819"}
Sep 09 15:20:13.886869 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.886842    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h7ljr" event={"ID":"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7","Type":"ContainerStarted","Data":"a964f6d795aad217af49897fab9585c5c6c62b43cd531684c18fe34cd8388709"}
Sep 09 15:20:13.892508 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.892486    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78" event={"ID":"44d88cca-0066-4eb8-8cb2-39c71959ec98","Type":"ContainerStarted","Data":"81e4b791c44be43fe9e1a136a06feaee44269364198903a4250121604b3792d7"}
Sep 09 15:20:13.894849 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.894824    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qn7pt" event={"ID":"0b0bf69a-0e36-446f-8748-a84c778fe36e","Type":"ContainerStarted","Data":"09808586516e20dc7c600c97c3d918da681edc3a563f39a9c9e3f9f04e99c6b7"}
Sep 09 15:20:13.897592 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.897568    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2wkxv" event={"ID":"879098a0-7a7e-4b71-b0ee-b7c759162e26","Type":"ContainerStarted","Data":"fe1feda9dc864c2f89c862e82f5a1cbd1b757090b451e56112fa33a54348180b"}
Sep 09 15:20:13.899674 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:13.899651    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42r95" event={"ID":"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb","Type":"ContainerStarted","Data":"090bdf9199d630684bb1be375a374faabae1f585502d5c25ac681eaad7c962f8"}
Sep 09 15:20:14.129347 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:14.129100    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 15:20:14.336569 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:14.336529    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs\") pod \"network-metrics-daemon-nhtcj\" (UID: \"7052f7a0-16d5-4115-82ae-34f07e9db9f9\") " pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:14.336747 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:14.336731    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:14.336817 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:14.336798    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs podName:7052f7a0-16d5-4115-82ae-34f07e9db9f9 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:16.336779719 +0000 UTC m=+5.978821708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs") pod "network-metrics-daemon-nhtcj" (UID: "7052f7a0-16d5-4115-82ae-34f07e9db9f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:14.438154 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:14.437440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret\") pod \"global-pull-secret-syncer-h4r6t\" (UID: \"560288f5-fbb4-4d6b-84bd-0987afe4c482\") " pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:14.438154 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:14.437656    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:14.438154 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:14.437736    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret podName:560288f5-fbb4-4d6b-84bd-0987afe4c482 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:16.437716038 +0000 UTC m=+6.079758018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret") pod "global-pull-secret-syncer-h4r6t" (UID: "560288f5-fbb4-4d6b-84bd-0987afe4c482") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:14.538667 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:14.538633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nms7\" (UniqueName: \"kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7\") pod \"network-check-target-7x79b\" (UID: \"b1406d46-d991-410d-bc6d-be16046a2fd5\") " pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:14.538842 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:14.538822    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:20:14.538881 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:14.538847    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:20:14.538881 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:14.538859    2578 projected.go:196] Error preparing data for projected volume kube-api-access-6nms7 for pod openshift-network-diagnostics/network-check-target-7x79b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:14.538966 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:14.538921    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7 podName:b1406d46-d991-410d-bc6d-be16046a2fd5 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:16.538903675 +0000 UTC m=+6.180945665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6nms7" (UniqueName: "kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7") pod "network-check-target-7x79b" (UID: "b1406d46-d991-410d-bc6d-be16046a2fd5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:14.840277 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:14.840177    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 15:15:12 +0000 UTC" deadline="2028-06-16 13:24:53.12940908 +0000 UTC"
Sep 09 15:20:14.840277 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:14.840215    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15502h4m38.289199369s"
Sep 09 15:20:14.867848 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:14.864208    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:14.867848 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:14.864258    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:14.867848 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:14.864406    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4r6t" podUID="560288f5-fbb4-4d6b-84bd-0987afe4c482"
Sep 09 15:20:14.867848 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:14.864902    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7x79b" podUID="b1406d46-d991-410d-bc6d-be16046a2fd5"
Sep 09 15:20:14.867848 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:14.864995    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:14.867848 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:14.865138    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:20:16.352340 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:16.352281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs\") pod \"network-metrics-daemon-nhtcj\" (UID: \"7052f7a0-16d5-4115-82ae-34f07e9db9f9\") " pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:16.352783 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:16.352415    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:16.352783 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:16.352482    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs podName:7052f7a0-16d5-4115-82ae-34f07e9db9f9 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:20.352460784 +0000 UTC m=+9.994502774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs") pod "network-metrics-daemon-nhtcj" (UID: "7052f7a0-16d5-4115-82ae-34f07e9db9f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:16.453348 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:16.453305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret\") pod \"global-pull-secret-syncer-h4r6t\" (UID: \"560288f5-fbb4-4d6b-84bd-0987afe4c482\") " pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:16.453536 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:16.453497    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:16.453597 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:16.453560    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret podName:560288f5-fbb4-4d6b-84bd-0987afe4c482 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:20.453540451 +0000 UTC m=+10.095582433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret") pod "global-pull-secret-syncer-h4r6t" (UID: "560288f5-fbb4-4d6b-84bd-0987afe4c482") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:16.554711 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:16.554671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nms7\" (UniqueName: \"kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7\") pod \"network-check-target-7x79b\" (UID: \"b1406d46-d991-410d-bc6d-be16046a2fd5\") " pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:16.554954 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:16.554921    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:20:16.555036 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:16.554965    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:20:16.555036 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:16.554994    2578 projected.go:196] Error preparing data for projected volume kube-api-access-6nms7 for pod openshift-network-diagnostics/network-check-target-7x79b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:16.555146 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:16.555081    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7 podName:b1406d46-d991-410d-bc6d-be16046a2fd5 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:20.555058148 +0000 UTC m=+10.197100119 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6nms7" (UniqueName: "kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7") pod "network-check-target-7x79b" (UID: "b1406d46-d991-410d-bc6d-be16046a2fd5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:16.862868 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:16.862830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:16.863066 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:16.862948    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4r6t" podUID="560288f5-fbb4-4d6b-84bd-0987afe4c482"
Sep 09 15:20:16.863132 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:16.863066    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:16.863199 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:16.863177    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7x79b" podUID="b1406d46-d991-410d-bc6d-be16046a2fd5"
Sep 09 15:20:16.863256 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:16.863236    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:16.863341 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:16.863322    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:20:18.861778 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:18.861733    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:18.862242 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:18.861866    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:20:18.862242 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:18.861925    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:18.862242 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:18.862048    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7x79b" podUID="b1406d46-d991-410d-bc6d-be16046a2fd5"
Sep 09 15:20:18.862242 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:18.862086    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:18.862242 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:18.862143    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4r6t" podUID="560288f5-fbb4-4d6b-84bd-0987afe4c482"
Sep 09 15:20:20.388246 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:20.388214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs\") pod \"network-metrics-daemon-nhtcj\" (UID: \"7052f7a0-16d5-4115-82ae-34f07e9db9f9\") " pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:20.388567 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:20.388348    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:20.388567 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:20.388394    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs podName:7052f7a0-16d5-4115-82ae-34f07e9db9f9 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:28.388381867 +0000 UTC m=+18.030423848 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs") pod "network-metrics-daemon-nhtcj" (UID: "7052f7a0-16d5-4115-82ae-34f07e9db9f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:20.488561 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:20.488524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret\") pod \"global-pull-secret-syncer-h4r6t\" (UID: \"560288f5-fbb4-4d6b-84bd-0987afe4c482\") " pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:20.488720 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:20.488700    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:20.488775 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:20.488760    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret podName:560288f5-fbb4-4d6b-84bd-0987afe4c482 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:28.488742185 +0000 UTC m=+18.130784166 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret") pod "global-pull-secret-syncer-h4r6t" (UID: "560288f5-fbb4-4d6b-84bd-0987afe4c482") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:20.589688 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:20.589047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nms7\" (UniqueName: \"kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7\") pod \"network-check-target-7x79b\" (UID: \"b1406d46-d991-410d-bc6d-be16046a2fd5\") " pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:20.589688 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:20.589256    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:20:20.589688 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:20.589277    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:20:20.589688 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:20.589289    2578 projected.go:196] Error preparing data for projected volume kube-api-access-6nms7 for pod openshift-network-diagnostics/network-check-target-7x79b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:20.589688 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:20.589347    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7 podName:b1406d46-d991-410d-bc6d-be16046a2fd5 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:28.589328841 +0000 UTC m=+18.231370828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6nms7" (UniqueName: "kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7") pod "network-check-target-7x79b" (UID: "b1406d46-d991-410d-bc6d-be16046a2fd5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:20.864740 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:20.864268    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:20.864740 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:20.864269    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:20.864740 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:20.864397    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4r6t" podUID="560288f5-fbb4-4d6b-84bd-0987afe4c482"
Sep 09 15:20:20.864740 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:20.864274    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:20.864740 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:20.864505    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:20:20.864740 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:20.864586    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7x79b" podUID="b1406d46-d991-410d-bc6d-be16046a2fd5"
Sep 09 15:20:22.860713 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:22.860680    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:22.861186 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:22.860838    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4r6t" podUID="560288f5-fbb4-4d6b-84bd-0987afe4c482"
Sep 09 15:20:22.861294 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:22.861275    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:22.861406 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:22.861376    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7x79b" podUID="b1406d46-d991-410d-bc6d-be16046a2fd5"
Sep 09 15:20:22.861464 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:22.861450    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:22.861562 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:22.861542    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:20:24.861087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:24.861051    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:24.861087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:24.861101    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:24.861728 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:24.861067    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:24.861728 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:24.861215    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7x79b" podUID="b1406d46-d991-410d-bc6d-be16046a2fd5"
Sep 09 15:20:24.861728 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:24.861312    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:20:24.861728 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:24.861394    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4r6t" podUID="560288f5-fbb4-4d6b-84bd-0987afe4c482"
Sep 09 15:20:26.861034 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:26.860995    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:26.861474 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:26.861008    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:26.861474 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:26.861127    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4r6t" podUID="560288f5-fbb4-4d6b-84bd-0987afe4c482"
Sep 09 15:20:26.861474 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:26.861146    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:26.861474 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:26.861232    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7x79b" podUID="b1406d46-d991-410d-bc6d-be16046a2fd5"
Sep 09 15:20:26.861474 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:26.861315    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:20:28.442872 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:28.442831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs\") pod \"network-metrics-daemon-nhtcj\" (UID: \"7052f7a0-16d5-4115-82ae-34f07e9db9f9\") " pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:28.443387 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:28.442957    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:28.443387 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:28.443033    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs podName:7052f7a0-16d5-4115-82ae-34f07e9db9f9 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:44.443018601 +0000 UTC m=+34.085060568 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs") pod "network-metrics-daemon-nhtcj" (UID: "7052f7a0-16d5-4115-82ae-34f07e9db9f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:28.543826 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:28.543795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret\") pod \"global-pull-secret-syncer-h4r6t\" (UID: \"560288f5-fbb4-4d6b-84bd-0987afe4c482\") " pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:28.544076 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:28.543921    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:28.544076 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:28.544012    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret podName:560288f5-fbb4-4d6b-84bd-0987afe4c482 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:44.54397288 +0000 UTC m=+34.186014850 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret") pod "global-pull-secret-syncer-h4r6t" (UID: "560288f5-fbb4-4d6b-84bd-0987afe4c482") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:28.644907 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:28.644871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nms7\" (UniqueName: \"kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7\") pod \"network-check-target-7x79b\" (UID: \"b1406d46-d991-410d-bc6d-be16046a2fd5\") " pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:28.645103 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:28.645036    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:20:28.645103 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:28.645064    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:20:28.645103 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:28.645077    2578 projected.go:196] Error preparing data for projected volume kube-api-access-6nms7 for pod openshift-network-diagnostics/network-check-target-7x79b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:28.645263 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:28.645141    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7 podName:b1406d46-d991-410d-bc6d-be16046a2fd5 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:44.645123932 +0000 UTC m=+34.287165901 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6nms7" (UniqueName: "kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7") pod "network-check-target-7x79b" (UID: "b1406d46-d991-410d-bc6d-be16046a2fd5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:28.860620 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:28.860538    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:28.860774 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:28.860549    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:28.860774 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:28.860674    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7x79b" podUID="b1406d46-d991-410d-bc6d-be16046a2fd5"
Sep 09 15:20:28.860774 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:28.860549    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:28.860774 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:28.860745    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4r6t" podUID="560288f5-fbb4-4d6b-84bd-0987afe4c482"
Sep 09 15:20:28.860969 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:28.860855    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:20:30.861202 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:30.861170    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:30.861649 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:30.861280    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4r6t" podUID="560288f5-fbb4-4d6b-84bd-0987afe4c482"
Sep 09 15:20:30.861649 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:30.861359    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:30.861649 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:30.861480    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:20:30.861649 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:30.861527    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:30.861649 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:30.861602    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7x79b" podUID="b1406d46-d991-410d-bc6d-be16046a2fd5"
Sep 09 15:20:31.950622 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:31.950418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h7ljr" event={"ID":"fd5bf48e-69b5-4bfc-8a0b-4d9d6a8a6bc7","Type":"ContainerStarted","Data":"43953b6acea33cc41995a8fdae391660b7f23c13b3c2c36dd9daeaae681eef60"}
Sep 09 15:20:31.961882 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:31.961853    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:20:31.964062 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:31.963695    2578 generic.go:356] "Generic (PLEG): container finished" podID="8e8d5b7a-7aa0-40dd-951e-76e34bf395cb" containerID="7bdf4155545901f89caa8b1f8934305b2edabd1eb80078cfb6a41adcc46685d7" exitCode=1
Sep 09 15:20:31.964062 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:31.963788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42r95" event={"ID":"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb","Type":"ContainerStarted","Data":"d91b8fb453de13eaafd021ed5839c3a8605cbde91c52d277cdc6a9b38d994433"}
Sep 09 15:20:31.964062 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:31.963873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42r95" event={"ID":"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb","Type":"ContainerStarted","Data":"9ba939d3cabcef1339b1003e3c03a84908392ce0d0eda2d27522fa50e6986295"}
Sep 09 15:20:31.964062 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:31.963929    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42r95" event={"ID":"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb","Type":"ContainerDied","Data":"7bdf4155545901f89caa8b1f8934305b2edabd1eb80078cfb6a41adcc46685d7"}
Sep 09 15:20:31.964062 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:31.964005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42r95" event={"ID":"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb","Type":"ContainerStarted","Data":"fbe6f0caeff6e632ce2d80eee22d6afa32f469678b826f2cf3b0f9f452492b56"}
Sep 09 15:20:31.965248 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:31.965193    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h7ljr" podStartSLOduration=2.695395043 podStartE2EDuration="20.965175674s" totalImagesPullingTime="18.269780631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:31.964622011 +0000 UTC m=+21.606663999" watchObservedRunningTime="2026-09-09 15:20:31.965175674 +0000 UTC m=+21.607217664"
Sep 09 15:20:31.967910 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:31.967885    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal" event={"ID":"f5569ed1edc68219f13a87c1afb087c8","Type":"ContainerStarted","Data":"f7e94d99ac519014bc20a6c9f55942931be3b5a0b8b9c1966d40f868f755a17f"}
Sep 09 15:20:32.860825 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:32.860588    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:32.861079 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:32.860625    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:32.861079 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:32.860925    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4r6t" podUID="560288f5-fbb4-4d6b-84bd-0987afe4c482"
Sep 09 15:20:32.861079 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:32.860641    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:32.861079 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:32.860965    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7x79b" podUID="b1406d46-d991-410d-bc6d-be16046a2fd5"
Sep 09 15:20:32.861304 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:32.861080    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:20:32.971716 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:32.971645    2578 generic.go:356] "Generic (PLEG): container finished" podID="879098a0-7a7e-4b71-b0ee-b7c759162e26" containerID="6b17b5f465f184fbf230d9bcbedeace524b0e986b21ea92b4ba223b1aaeec9c9" exitCode=0
Sep 09 15:20:32.972382 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:32.971729    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2wkxv" event={"ID":"879098a0-7a7e-4b71-b0ee-b7c759162e26","Type":"ContainerDied","Data":"6b17b5f465f184fbf230d9bcbedeace524b0e986b21ea92b4ba223b1aaeec9c9"}
Sep 09 15:20:32.977680 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:32.977659    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:20:32.978108 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:32.978081    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42r95" event={"ID":"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb","Type":"ContainerStarted","Data":"a5eeaff0f5378aa273569d50c5cbac2dda05bcb944322ad54584051a21a2cb92"}
Sep 09 15:20:32.978225 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:32.978192    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42r95" event={"ID":"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb","Type":"ContainerStarted","Data":"fb24e4587da240303878e57510d127c458ccef2518e32748eedffd97f0a6c693"}
Sep 09 15:20:32.979483 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:32.979462    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v2hbm" event={"ID":"9fc01e4b-4a6f-47f0-8220-ca62e74d7250","Type":"ContainerStarted","Data":"36990e76bbff37964398e5a78315135691342766e6a855be4dd30e6cebd96a47"}
Sep 09 15:20:32.981283 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:32.981251    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7kt4h" event={"ID":"588589aa-effd-4129-96af-6180f4a1b5d8","Type":"ContainerStarted","Data":"98ee1ee19302c91d1f986476e0b467d069001cbfb3f7264c22a95034661cf8e9"}
Sep 09 15:20:32.982585 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:32.982566    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fvk5x" event={"ID":"bd6109fe-7e99-4d98-87d0-8a6fc95d6938","Type":"ContainerStarted","Data":"c4c6793820c01457ff887d50826f663b245a97cac720757442e6e7dca2e634b7"}
Sep 09 15:20:32.983904 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:32.983889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ws5dd" event={"ID":"adc5a00c-c7b6-42c7-9f72-1c86bcb50e38","Type":"ContainerStarted","Data":"ee162fc9e75645a4c31714f1ea07ef4f627a0dca1c4e1db10c030f7c61e95f10"}
Sep 09 15:20:32.985189 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:32.985172    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78" event={"ID":"44d88cca-0066-4eb8-8cb2-39c71959ec98","Type":"ContainerStarted","Data":"6b858cac01b55826a448f00b30a32a9cab71bb846b8a940ca0b83de04b5c4505"}
Sep 09 15:20:32.986420 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:32.986401    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qn7pt" event={"ID":"0b0bf69a-0e36-446f-8748-a84c778fe36e","Type":"ContainerStarted","Data":"45f6250e7d40edc6bcd88be5a3b1b69ae6de1f39edae178686e3661ead351935"}
Sep 09 15:20:32.987841 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:32.987819    2578 generic.go:356] "Generic (PLEG): container finished" podID="d59a6b49d2c887c886a8231821e4f270" containerID="34acebcf7a63f3be6a39899e1f2bf6a2aa68c943ddc8887d8afc6d9a8e7a2dd2" exitCode=0
Sep 09 15:20:32.987919 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:32.987897    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal" event={"ID":"d59a6b49d2c887c886a8231821e4f270","Type":"ContainerDied","Data":"34acebcf7a63f3be6a39899e1f2bf6a2aa68c943ddc8887d8afc6d9a8e7a2dd2"}
Sep 09 15:20:32.993416 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:32.993374    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.ec2.internal" podStartSLOduration=20.993358615 podStartE2EDuration="20.993358615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:31.981856105 +0000 UTC m=+21.623898095" watchObservedRunningTime="2026-09-09 15:20:32.993358615 +0000 UTC m=+22.635400605"
Sep 09 15:20:33.006315 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:33.006281    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fvk5x" podStartSLOduration=3.16778209 podStartE2EDuration="22.006266636s" totalImagesPullingTime="18.838484546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:33.005849344 +0000 UTC m=+22.647891334" watchObservedRunningTime="2026-09-09 15:20:33.006266636 +0000 UTC m=+22.648308625"
Sep 09 15:20:33.019711 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:33.019671    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v2hbm" podStartSLOduration=3.783001715 podStartE2EDuration="22.019660203s" totalImagesPullingTime="18.236658488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:33.018799205 +0000 UTC m=+22.660841195" watchObservedRunningTime="2026-09-09 15:20:33.019660203 +0000 UTC m=+22.661702184"
Sep 09 15:20:33.031746 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:33.031707    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qn7pt" podStartSLOduration=8.834012960999999 podStartE2EDuration="23.031693312s" totalImagesPullingTime="14.197680351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:33.031114576 +0000 UTC m=+22.673156591" watchObservedRunningTime="2026-09-09 15:20:33.031693312 +0000 UTC m=+22.673735305"
Sep 09 15:20:33.043808 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:33.043765    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ws5dd" podStartSLOduration=3.775004738 podStartE2EDuration="22.043750163s" totalImagesPullingTime="18.268745425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:33.043514186 +0000 UTC m=+22.685556177" watchObservedRunningTime="2026-09-09 15:20:33.043750163 +0000 UTC m=+22.685792153"
Sep 09 15:20:33.057034 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:33.056974    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7kt4h" podStartSLOduration=3.817437079 podStartE2EDuration="22.056964383s" totalImagesPullingTime="18.239527304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:33.056755637 +0000 UTC m=+22.698797628" watchObservedRunningTime="2026-09-09 15:20:33.056964383 +0000 UTC m=+22.699006372"
Sep 09 15:20:33.993336 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:33.993291    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal" event={"ID":"d59a6b49d2c887c886a8231821e4f270","Type":"ContainerStarted","Data":"8dfdb2e6325c56b18d1dcaccd2a1af6614f8651a9a4a4308612af1753b608016"}
Sep 09 15:20:34.008729 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:34.008687    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.ec2.internal" podStartSLOduration=22.008670169 podStartE2EDuration="22.008670169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:34.008059004 +0000 UTC m=+23.650100992" watchObservedRunningTime="2026-09-09 15:20:34.008670169 +0000 UTC m=+23.650712157"
Sep 09 15:20:34.164253 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:34.164059    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 15:20:34.783234 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:34.783134    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T15:20:34.164228462Z","UUID":"33258faf-6e62-4934-9e41-775c2a058527","Handler":null,"Name":"","Endpoint":""}
Sep 09 15:20:34.784964 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:34.784943    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 15:20:34.785106 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:34.784992    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 15:20:34.860539 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:34.860510    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:34.860692 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:34.860510    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:34.860763 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:34.860733    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:20:34.860763 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:34.860612    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7x79b" podUID="b1406d46-d991-410d-bc6d-be16046a2fd5"
Sep 09 15:20:34.860763 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:34.860512    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:34.860943 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:34.860835    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4r6t" podUID="560288f5-fbb4-4d6b-84bd-0987afe4c482"
Sep 09 15:20:34.998562 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:34.998523    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78" event={"ID":"44d88cca-0066-4eb8-8cb2-39c71959ec98","Type":"ContainerStarted","Data":"62b0f226355fd6788cbb23b83119917e0dd4c606e600ee801189c0d0c924c44a"}
Sep 09 15:20:35.002709 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:35.002683    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:20:35.003446 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:35.003402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42r95" event={"ID":"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb","Type":"ContainerStarted","Data":"5382afeb31f15b93236958d9c78ed8ae6f6b1e63770b1aedf7024ad1a4094f79"}
Sep 09 15:20:35.666003 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:35.665905    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7kt4h"
Sep 09 15:20:35.666558 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:35.666542    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7kt4h"
Sep 09 15:20:36.008071 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:36.007991    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78" event={"ID":"44d88cca-0066-4eb8-8cb2-39c71959ec98","Type":"ContainerStarted","Data":"ef3b1aaf2f8e3e9787d85124d1eeca1e84edcb7e61e12dd8d18063cac21bb948"}
Sep 09 15:20:36.008496 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:36.008153    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7kt4h"
Sep 09 15:20:36.008958 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:36.008938    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7kt4h"
Sep 09 15:20:36.027050 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:36.027011    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzd78" podStartSLOduration=4.63855221 podStartE2EDuration="26.026995302s" totalImagesPullingTime="21.388443092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:36.026878517 +0000 UTC m=+25.668920507" watchObservedRunningTime="2026-09-09 15:20:36.026995302 +0000 UTC m=+25.669037300"
Sep 09 15:20:36.861372 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:36.861340    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:36.861561 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:36.861341    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:36.861561 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:36.861457    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4r6t" podUID="560288f5-fbb4-4d6b-84bd-0987afe4c482"
Sep 09 15:20:36.861561 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:36.861516    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7x79b" podUID="b1406d46-d991-410d-bc6d-be16046a2fd5"
Sep 09 15:20:36.861561 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:36.861341    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:36.861772 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:36.861684    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:20:38.013596 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:38.013374    2578 generic.go:356] "Generic (PLEG): container finished" podID="879098a0-7a7e-4b71-b0ee-b7c759162e26" containerID="88fb444d2f35f64ba6174f6bedfe293584c2b9135833cd848e69aec9ed2191bb" exitCode=0
Sep 09 15:20:38.014019 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:38.013452    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2wkxv" event={"ID":"879098a0-7a7e-4b71-b0ee-b7c759162e26","Type":"ContainerDied","Data":"88fb444d2f35f64ba6174f6bedfe293584c2b9135833cd848e69aec9ed2191bb"}
Sep 09 15:20:38.017308 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:38.017220    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:20:38.017678 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:38.017655    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42r95" event={"ID":"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb","Type":"ContainerStarted","Data":"693b3aededf5da43d466d5d810088ef45909c11def1c035225718093747faf0f"}
Sep 09 15:20:38.017916 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:38.017895    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:38.018125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:38.018106    2578 scope.go:117] "RemoveContainer" containerID="7bdf4155545901f89caa8b1f8934305b2edabd1eb80078cfb6a41adcc46685d7"
Sep 09 15:20:38.033562 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:38.033545    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:38.860562 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:38.860529    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:38.860643 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:38.860529    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:38.860709 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:38.860647    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:38.860709 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:38.860639    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4r6t" podUID="560288f5-fbb4-4d6b-84bd-0987afe4c482"
Sep 09 15:20:38.860801 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:38.860781    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:20:38.860922 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:38.860899    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7x79b" podUID="b1406d46-d991-410d-bc6d-be16046a2fd5"
Sep 09 15:20:39.021955 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:39.021873    2578 generic.go:356] "Generic (PLEG): container finished" podID="879098a0-7a7e-4b71-b0ee-b7c759162e26" containerID="4ed0f4076f42407337c1a3d026897474f21cc630632c2d5e3aee40e81350d266" exitCode=0
Sep 09 15:20:39.021955 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:39.021922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2wkxv" event={"ID":"879098a0-7a7e-4b71-b0ee-b7c759162e26","Type":"ContainerDied","Data":"4ed0f4076f42407337c1a3d026897474f21cc630632c2d5e3aee40e81350d266"}
Sep 09 15:20:39.026198 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:39.026178    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:20:39.026573 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:39.026553    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42r95" event={"ID":"8e8d5b7a-7aa0-40dd-951e-76e34bf395cb","Type":"ContainerStarted","Data":"05fc356d92cfcdcade3827eec6ada59f374ef3054363181628b3aed7ee6ed03f"}
Sep 09 15:20:39.026824 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:39.026796    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:39.026913 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:39.026881    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:39.042938 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:39.042915    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:20:39.068271 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:39.068229    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-42r95" podStartSLOduration=10.669806337 podStartE2EDuration="29.06821615s" totalImagesPullingTime="18.398409813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:39.0672525 +0000 UTC m=+28.709294491" watchObservedRunningTime="2026-09-09 15:20:39.06821615 +0000 UTC m=+28.710258138"
Sep 09 15:20:39.571124 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:39.571028    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h4r6t"]
Sep 09 15:20:39.571333 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:39.571236    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:39.571398 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:39.571364    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4r6t" podUID="560288f5-fbb4-4d6b-84bd-0987afe4c482"
Sep 09 15:20:39.571778 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:39.571758    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7x79b"]
Sep 09 15:20:39.571865 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:39.571845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:39.571937 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:39.571920    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7x79b" podUID="b1406d46-d991-410d-bc6d-be16046a2fd5"
Sep 09 15:20:39.582337 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:39.582316    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nhtcj"]
Sep 09 15:20:39.582436 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:39.582393    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:39.582475 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:39.582462    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:20:40.030802 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:40.030770    2578 generic.go:356] "Generic (PLEG): container finished" podID="879098a0-7a7e-4b71-b0ee-b7c759162e26" containerID="ba92388a10760f81268fd2e0fc6b73eb37d6938b9a078d3d752cd86b9ce5a053" exitCode=0
Sep 09 15:20:40.031305 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:40.030840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2wkxv" event={"ID":"879098a0-7a7e-4b71-b0ee-b7c759162e26","Type":"ContainerDied","Data":"ba92388a10760f81268fd2e0fc6b73eb37d6938b9a078d3d752cd86b9ce5a053"}
Sep 09 15:20:40.862474 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:40.862254    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:40.862633 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:40.862326    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:40.862633 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:40.862566    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4r6t" podUID="560288f5-fbb4-4d6b-84bd-0987afe4c482"
Sep 09 15:20:40.862756 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:40.862355    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:40.862756 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:40.862648    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7x79b" podUID="b1406d46-d991-410d-bc6d-be16046a2fd5"
Sep 09 15:20:40.862871 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:40.862779    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:20:42.861018 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:42.860991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:42.861018 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:42.861009    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:42.861524 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:42.861044    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:42.861524 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:42.861132    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4r6t" podUID="560288f5-fbb4-4d6b-84bd-0987afe4c482"
Sep 09 15:20:42.861524 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:42.861206    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7x79b" podUID="b1406d46-d991-410d-bc6d-be16046a2fd5"
Sep 09 15:20:42.861524 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:42.861299    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:20:43.673933 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.673901    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-183.ec2.internal" event="NodeReady"
Sep 09 15:20:43.674114 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.674070    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 15:20:43.706052 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.706024    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-62m26"]
Sep 09 15:20:43.745166 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.745135    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8945c7fd4-h4fdq"]
Sep 09 15:20:43.745320 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.745270    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:20:43.748404 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.748365    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hz8wm\""
Sep 09 15:20:43.748569 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.748479    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 09 15:20:43.748668 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.748653    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 09 15:20:43.754420 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.754391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.754519 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.754428    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-62m26"]
Sep 09 15:20:43.754519 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.754485    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rdv4c"]
Sep 09 15:20:43.757112 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.757093    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 15:20:43.757224 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.757194    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pzhsz\""
Sep 09 15:20:43.757708 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.757688    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 15:20:43.757967 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.757949    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 15:20:43.760537 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.760159    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j58rw"]
Sep 09 15:20:43.760537 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.760279    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:20:43.763732 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.762943    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 15:20:43.763732 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.763042    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 15:20:43.763732 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.763080    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jnqpd\""
Sep 09 15:20:43.763732 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.763603    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 15:20:43.768158 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.768140    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8945c7fd4-h4fdq"]
Sep 09 15:20:43.768247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.768168    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rdv4c"]
Sep 09 15:20:43.768247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.768184    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j58rw"]
Sep 09 15:20:43.768355 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.768251    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j58rw"
Sep 09 15:20:43.770376 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.770357    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mczr8\""
Sep 09 15:20:43.770627 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.770377    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 15:20:43.770627 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.770421    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 15:20:43.770751 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.770363    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 15:20:43.850625 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.850577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-62m26\" (UID: \"18973d3c-7fa4-4b71-ade6-1cd976413494\") " pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:20:43.850625 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.850633    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4637ce97-9134-469e-9b3c-388508a3858b-image-registry-private-configuration\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.850843 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.850649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqz8v\" (UniqueName: \"kubernetes.io/projected/ed80dd24-f513-4215-9f41-c33847a78c80-kube-api-access-gqz8v\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:20:43.850843 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.850731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4637ce97-9134-469e-9b3c-388508a3858b-registry-certificates\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.850843 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.850779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.850843 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.850810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ed80dd24-f513-4215-9f41-c33847a78c80-config-volume\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:20:43.850843 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.850839    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:20:43.851086 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.850853    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/18973d3c-7fa4-4b71-ade6-1cd976413494-nginx-conf\") pod \"networking-console-plugin-79977f9844-62m26\" (UID: \"18973d3c-7fa4-4b71-ade6-1cd976413494\") " pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:20:43.851086 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.850935    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4637ce97-9134-469e-9b3c-388508a3858b-trusted-ca\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.851086 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.850967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed80dd24-f513-4215-9f41-c33847a78c80-tmp-dir\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:20:43.851086 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.851003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b5c31a0-d5d8-4e50-ad6b-e0697ec8c073-cert\") pod \"ingress-canary-j58rw\" (UID: \"7b5c31a0-d5d8-4e50-ad6b-e0697ec8c073\") " pod="openshift-ingress-canary/ingress-canary-j58rw"
Sep 09 15:20:43.851086 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.851020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-bound-sa-token\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.851086 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.851077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4637ce97-9134-469e-9b3c-388508a3858b-ca-trust-extracted\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.851306 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.851102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fgfs\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-kube-api-access-9fgfs\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.851306 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.851127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl6ht\" (UniqueName: \"kubernetes.io/projected/7b5c31a0-d5d8-4e50-ad6b-e0697ec8c073-kube-api-access-hl6ht\") pod \"ingress-canary-j58rw\" (UID: \"7b5c31a0-d5d8-4e50-ad6b-e0697ec8c073\") " pod="openshift-ingress-canary/ingress-canary-j58rw"
Sep 09 15:20:43.851306 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.851152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4637ce97-9134-469e-9b3c-388508a3858b-installation-pull-secrets\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.861790 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.861762    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:20:43.944192 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.944157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:20:43.952094 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.952062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4637ce97-9134-469e-9b3c-388508a3858b-ca-trust-extracted\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.952094 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.952093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fgfs\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-kube-api-access-9fgfs\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.952285 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.952119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl6ht\" (UniqueName: \"kubernetes.io/projected/7b5c31a0-d5d8-4e50-ad6b-e0697ec8c073-kube-api-access-hl6ht\") pod \"ingress-canary-j58rw\" (UID: \"7b5c31a0-d5d8-4e50-ad6b-e0697ec8c073\") " pod="openshift-ingress-canary/ingress-canary-j58rw"
Sep 09 15:20:43.952341 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.952328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4637ce97-9134-469e-9b3c-388508a3858b-installation-pull-secrets\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.952397 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.952361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-62m26\" (UID: \"18973d3c-7fa4-4b71-ade6-1cd976413494\") " pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:20:43.952450 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.952407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4637ce97-9134-469e-9b3c-388508a3858b-image-registry-private-configuration\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.952450 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.952436    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqz8v\" (UniqueName: \"kubernetes.io/projected/ed80dd24-f513-4215-9f41-c33847a78c80-kube-api-access-gqz8v\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:20:43.952599 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.952480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4637ce97-9134-469e-9b3c-388508a3858b-registry-certificates\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.952599 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.952508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.952599 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.952531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ed80dd24-f513-4215-9f41-c33847a78c80-config-volume\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:20:43.952599 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.952540    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4637ce97-9134-469e-9b3c-388508a3858b-ca-trust-extracted\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.952599 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.952558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:20:43.952599 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.952584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/18973d3c-7fa4-4b71-ade6-1cd976413494-nginx-conf\") pod \"networking-console-plugin-79977f9844-62m26\" (UID: \"18973d3c-7fa4-4b71-ade6-1cd976413494\") " pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:20:43.952887 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:43.952620    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 15:20:43.952887 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.952641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4637ce97-9134-469e-9b3c-388508a3858b-trusted-ca\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.952887 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:43.952675    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert podName:18973d3c-7fa4-4b71-ade6-1cd976413494 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:44.452656031 +0000 UTC m=+34.094698013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-62m26" (UID: "18973d3c-7fa4-4b71-ade6-1cd976413494") : secret "networking-console-plugin-cert" not found
Sep 09 15:20:43.952887 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.952695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed80dd24-f513-4215-9f41-c33847a78c80-tmp-dir\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:20:43.952887 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.952727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b5c31a0-d5d8-4e50-ad6b-e0697ec8c073-cert\") pod \"ingress-canary-j58rw\" (UID: \"7b5c31a0-d5d8-4e50-ad6b-e0697ec8c073\") " pod="openshift-ingress-canary/ingress-canary-j58rw"
Sep 09 15:20:43.952887 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.952768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-bound-sa-token\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.953351 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:43.953329    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:20:43.953486 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:43.953431    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls podName:ed80dd24-f513-4215-9f41-c33847a78c80 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:44.453379928 +0000 UTC m=+34.095421895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls") pod "dns-default-rdv4c" (UID: "ed80dd24-f513-4215-9f41-c33847a78c80") : secret "dns-default-metrics-tls" not found
Sep 09 15:20:43.953678 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.953659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed80dd24-f513-4215-9f41-c33847a78c80-tmp-dir\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:20:43.953777 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.953676    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ed80dd24-f513-4215-9f41-c33847a78c80-config-volume\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:20:43.953777 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:43.953759    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 15:20:43.953777 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:43.953774    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8945c7fd4-h4fdq: secret "image-registry-tls" not found
Sep 09 15:20:43.953922 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:43.953825    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls podName:4637ce97-9134-469e-9b3c-388508a3858b nodeName:}" failed. No retries permitted until 2026-09-09 15:20:44.453809898 +0000 UTC m=+34.095851882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls") pod "image-registry-8945c7fd4-h4fdq" (UID: "4637ce97-9134-469e-9b3c-388508a3858b") : secret "image-registry-tls" not found
Sep 09 15:20:43.954002 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.953958    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/18973d3c-7fa4-4b71-ade6-1cd976413494-nginx-conf\") pod \"networking-console-plugin-79977f9844-62m26\" (UID: \"18973d3c-7fa4-4b71-ade6-1cd976413494\") " pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:20:43.954053 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.954004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4637ce97-9134-469e-9b3c-388508a3858b-trusted-ca\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.954261 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.954239    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4637ce97-9134-469e-9b3c-388508a3858b-registry-certificates\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.957727 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.957696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4637ce97-9134-469e-9b3c-388508a3858b-image-registry-private-configuration\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.957820 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.957727    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b5c31a0-d5d8-4e50-ad6b-e0697ec8c073-cert\") pod \"ingress-canary-j58rw\" (UID: \"7b5c31a0-d5d8-4e50-ad6b-e0697ec8c073\") " pod="openshift-ingress-canary/ingress-canary-j58rw"
Sep 09 15:20:43.958570 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.958550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4637ce97-9134-469e-9b3c-388508a3858b-installation-pull-secrets\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.960507 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.960484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fgfs\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-kube-api-access-9fgfs\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:43.960613 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.960511    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl6ht\" (UniqueName: \"kubernetes.io/projected/7b5c31a0-d5d8-4e50-ad6b-e0697ec8c073-kube-api-access-hl6ht\") pod \"ingress-canary-j58rw\" (UID: \"7b5c31a0-d5d8-4e50-ad6b-e0697ec8c073\") " pod="openshift-ingress-canary/ingress-canary-j58rw"
Sep 09 15:20:43.961793 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.961773    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqz8v\" (UniqueName: \"kubernetes.io/projected/ed80dd24-f513-4215-9f41-c33847a78c80-kube-api-access-gqz8v\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:20:43.961894 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:43.961873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-bound-sa-token\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:44.080678 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:44.080644    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j58rw"
Sep 09 15:20:44.456938 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:44.456901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:44.456938 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:44.456952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:20:44.457186 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:44.457053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs\") pod \"network-metrics-daemon-nhtcj\" (UID: \"7052f7a0-16d5-4115-82ae-34f07e9db9f9\") " pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:44.457186 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:44.457094    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 15:20:44.457186 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:44.457116    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8945c7fd4-h4fdq: secret "image-registry-tls" not found
Sep 09 15:20:44.457186 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:44.457139    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:20:44.457186 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:44.457100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-62m26\" (UID: \"18973d3c-7fa4-4b71-ade6-1cd976413494\") " pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:20:44.457186 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:44.457174    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 15:20:44.457186 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:44.457180    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls podName:4637ce97-9134-469e-9b3c-388508a3858b nodeName:}" failed. No retries permitted until 2026-09-09 15:20:45.457150219 +0000 UTC m=+35.099192209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls") pod "image-registry-8945c7fd4-h4fdq" (UID: "4637ce97-9134-469e-9b3c-388508a3858b") : secret "image-registry-tls" not found
Sep 09 15:20:44.457186 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:44.457203    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:44.457186 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:44.457211    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls podName:ed80dd24-f513-4215-9f41-c33847a78c80 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:45.457194581 +0000 UTC m=+35.099236566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls") pod "dns-default-rdv4c" (UID: "ed80dd24-f513-4215-9f41-c33847a78c80") : secret "dns-default-metrics-tls" not found
Sep 09 15:20:44.457524 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:44.457233    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert podName:18973d3c-7fa4-4b71-ade6-1cd976413494 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:45.457221651 +0000 UTC m=+35.099263619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-62m26" (UID: "18973d3c-7fa4-4b71-ade6-1cd976413494") : secret "networking-console-plugin-cert" not found
Sep 09 15:20:44.457524 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:44.457259    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs podName:7052f7a0-16d5-4115-82ae-34f07e9db9f9 nodeName:}" failed. No retries permitted until 2026-09-09 15:21:16.457246444 +0000 UTC m=+66.099288413 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs") pod "network-metrics-daemon-nhtcj" (UID: "7052f7a0-16d5-4115-82ae-34f07e9db9f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 15:20:44.557741 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:44.557657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret\") pod \"global-pull-secret-syncer-h4r6t\" (UID: \"560288f5-fbb4-4d6b-84bd-0987afe4c482\") " pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:44.557881 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:44.557813    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:44.557924 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:44.557884    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret podName:560288f5-fbb4-4d6b-84bd-0987afe4c482 nodeName:}" failed. No retries permitted until 2026-09-09 15:21:16.557865339 +0000 UTC m=+66.199907312 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret") pod "global-pull-secret-syncer-h4r6t" (UID: "560288f5-fbb4-4d6b-84bd-0987afe4c482") : object "kube-system"/"original-pull-secret" not registered
Sep 09 15:20:44.658567 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:44.658530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nms7\" (UniqueName: \"kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7\") pod \"network-check-target-7x79b\" (UID: \"b1406d46-d991-410d-bc6d-be16046a2fd5\") " pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:44.658714 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:44.658659    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 15:20:44.658714 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:44.658684    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 15:20:44.658714 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:44.658694    2578 projected.go:196] Error preparing data for projected volume kube-api-access-6nms7 for pod openshift-network-diagnostics/network-check-target-7x79b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:44.658835 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:44.658745    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7 podName:b1406d46-d991-410d-bc6d-be16046a2fd5 nodeName:}" failed. No retries permitted until 2026-09-09 15:21:16.658731322 +0000 UTC m=+66.300773306 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6nms7" (UniqueName: "kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7") pod "network-check-target-7x79b" (UID: "b1406d46-d991-410d-bc6d-be16046a2fd5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 15:20:44.860575 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:44.860546    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:20:44.860690 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:44.860548    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:20:44.860690 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:44.860685    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:20:44.863008 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:44.862992    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 15:20:44.863939 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:44.863902    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 15:20:44.864049 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:44.863941    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 15:20:44.864049 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:44.863953    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xdr6q\""
Sep 09 15:20:44.864049 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:44.863961    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7889l\""
Sep 09 15:20:44.864049 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:44.863998    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 15:20:44.887268 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:44.887242    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j58rw"]
Sep 09 15:20:44.897922 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:20:44.897730    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5c31a0_d5d8_4e50_ad6b_e0697ec8c073.slice/crio-1c62b8ec3e4656ba64b1bdef0623d46742b1120c00ab788235b963f57f937f6f WatchSource:0}: Error finding container 1c62b8ec3e4656ba64b1bdef0623d46742b1120c00ab788235b963f57f937f6f: Status 404 returned error can't find the container with id 1c62b8ec3e4656ba64b1bdef0623d46742b1120c00ab788235b963f57f937f6f
Sep 09 15:20:45.048464 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:45.048314    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2wkxv" event={"ID":"879098a0-7a7e-4b71-b0ee-b7c759162e26","Type":"ContainerStarted","Data":"3eadce4446117d7b91e60dccb1c181305bd11055706d01c8c82cb7e7c94ce975"}
Sep 09 15:20:45.049789 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:45.049763    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j58rw" event={"ID":"7b5c31a0-d5d8-4e50-ad6b-e0697ec8c073","Type":"ContainerStarted","Data":"1c62b8ec3e4656ba64b1bdef0623d46742b1120c00ab788235b963f57f937f6f"}
Sep 09 15:20:45.464389 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:45.464351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-62m26\" (UID: \"18973d3c-7fa4-4b71-ade6-1cd976413494\") " pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:20:45.464663 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:45.464413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:45.464663 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:45.464519    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 15:20:45.464663 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:45.464527    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 15:20:45.464663 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:45.464583    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:20:45.464663 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:45.464532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:20:45.464663 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:45.464535    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8945c7fd4-h4fdq: secret "image-registry-tls" not found
Sep 09 15:20:45.464663 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:45.464605    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert podName:18973d3c-7fa4-4b71-ade6-1cd976413494 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:47.464582985 +0000 UTC m=+37.106624971 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-62m26" (UID: "18973d3c-7fa4-4b71-ade6-1cd976413494") : secret "networking-console-plugin-cert" not found
Sep 09 15:20:45.464663 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:45.464642    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls podName:ed80dd24-f513-4215-9f41-c33847a78c80 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:47.464629958 +0000 UTC m=+37.106671924 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls") pod "dns-default-rdv4c" (UID: "ed80dd24-f513-4215-9f41-c33847a78c80") : secret "dns-default-metrics-tls" not found
Sep 09 15:20:45.464663 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:45.464656    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls podName:4637ce97-9134-469e-9b3c-388508a3858b nodeName:}" failed. No retries permitted until 2026-09-09 15:20:47.464650176 +0000 UTC m=+37.106692143 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls") pod "image-registry-8945c7fd4-h4fdq" (UID: "4637ce97-9134-469e-9b3c-388508a3858b") : secret "image-registry-tls" not found
Sep 09 15:20:46.055393 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:46.055360    2578 generic.go:356] "Generic (PLEG): container finished" podID="879098a0-7a7e-4b71-b0ee-b7c759162e26" containerID="3eadce4446117d7b91e60dccb1c181305bd11055706d01c8c82cb7e7c94ce975" exitCode=0
Sep 09 15:20:46.055393 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:46.055398    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2wkxv" event={"ID":"879098a0-7a7e-4b71-b0ee-b7c759162e26","Type":"ContainerDied","Data":"3eadce4446117d7b91e60dccb1c181305bd11055706d01c8c82cb7e7c94ce975"}
Sep 09 15:20:47.061611 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:47.061580    2578 generic.go:356] "Generic (PLEG): container finished" podID="879098a0-7a7e-4b71-b0ee-b7c759162e26" containerID="873d30e35b2bd57a67483f253d4f4b69414eff9291d68e9f3bf597b103b012aa" exitCode=0
Sep 09 15:20:47.062206 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:47.061653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2wkxv" event={"ID":"879098a0-7a7e-4b71-b0ee-b7c759162e26","Type":"ContainerDied","Data":"873d30e35b2bd57a67483f253d4f4b69414eff9291d68e9f3bf597b103b012aa"}
Sep 09 15:20:47.482664 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:47.482641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-62m26\" (UID: \"18973d3c-7fa4-4b71-ade6-1cd976413494\") " pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:20:47.482780 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:47.482688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:47.482780 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:47.482706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:20:47.482930 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:47.482796    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:20:47.482930 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:47.482808    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 15:20:47.482930 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:47.482812    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 15:20:47.482930 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:47.482834    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8945c7fd4-h4fdq: secret "image-registry-tls" not found
Sep 09 15:20:47.482930 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:47.482843    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls podName:ed80dd24-f513-4215-9f41-c33847a78c80 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:51.48283052 +0000 UTC m=+41.124872499 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls") pod "dns-default-rdv4c" (UID: "ed80dd24-f513-4215-9f41-c33847a78c80") : secret "dns-default-metrics-tls" not found
Sep 09 15:20:47.482930 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:47.482871    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert podName:18973d3c-7fa4-4b71-ade6-1cd976413494 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:51.482851253 +0000 UTC m=+41.124893240 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-62m26" (UID: "18973d3c-7fa4-4b71-ade6-1cd976413494") : secret "networking-console-plugin-cert" not found
Sep 09 15:20:47.482930 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:47.482892    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls podName:4637ce97-9134-469e-9b3c-388508a3858b nodeName:}" failed. No retries permitted until 2026-09-09 15:20:51.482881061 +0000 UTC m=+41.124923034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls") pod "image-registry-8945c7fd4-h4fdq" (UID: "4637ce97-9134-469e-9b3c-388508a3858b") : secret "image-registry-tls" not found
Sep 09 15:20:48.066663 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:48.066632    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2wkxv" event={"ID":"879098a0-7a7e-4b71-b0ee-b7c759162e26","Type":"ContainerStarted","Data":"e01075a78253218295ae525d049448575dacd08f07649a3a04c946c7f3229081"}
Sep 09 15:20:48.068069 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:48.068047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j58rw" event={"ID":"7b5c31a0-d5d8-4e50-ad6b-e0697ec8c073","Type":"ContainerStarted","Data":"f83f1ca38c3a33895b19979b4171546793734ee99045a3289ea2e414aed75150"}
Sep 09 15:20:48.086329 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:48.086278    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2wkxv" podStartSLOduration=8.648334377 podStartE2EDuration="38.086260905s" totalImagesPullingTime="29.437926528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:48.084965233 +0000 UTC m=+37.727007221" watchObservedRunningTime="2026-09-09 15:20:48.086260905 +0000 UTC m=+37.728302895"
Sep 09 15:20:48.098319 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:48.098279    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j58rw" podStartSLOduration=2.483588047 podStartE2EDuration="5.098267585s" totalImagesPullingTime="2.614679538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:20:48.097719688 +0000 UTC m=+37.739761674" watchObservedRunningTime="2026-09-09 15:20:48.098267585 +0000 UTC m=+37.740309566"
Sep 09 15:20:48.212593 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:48.212550    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:20:48.312764 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:48.312730    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:20:49.785269 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:49.785239    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:20:49.883198 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:49.883167    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:20:51.514361 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:51.514315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:51.514361 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:51.514360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:20:51.514772 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:51.514419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-62m26\" (UID: \"18973d3c-7fa4-4b71-ade6-1cd976413494\") " pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:20:51.514772 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:51.514485    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 15:20:51.514772 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:51.514503    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 15:20:51.514772 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:51.514510    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8945c7fd4-h4fdq: secret "image-registry-tls" not found
Sep 09 15:20:51.514772 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:51.514512    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:20:51.514772 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:51.514561    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert podName:18973d3c-7fa4-4b71-ade6-1cd976413494 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:59.514543456 +0000 UTC m=+49.156585423 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-62m26" (UID: "18973d3c-7fa4-4b71-ade6-1cd976413494") : secret "networking-console-plugin-cert" not found
Sep 09 15:20:51.514772 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:51.514577    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls podName:ed80dd24-f513-4215-9f41-c33847a78c80 nodeName:}" failed. No retries permitted until 2026-09-09 15:20:59.514570685 +0000 UTC m=+49.156612652 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls") pod "dns-default-rdv4c" (UID: "ed80dd24-f513-4215-9f41-c33847a78c80") : secret "dns-default-metrics-tls" not found
Sep 09 15:20:51.514772 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:51.514586    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls podName:4637ce97-9134-469e-9b3c-388508a3858b nodeName:}" failed. No retries permitted until 2026-09-09 15:20:59.514581257 +0000 UTC m=+49.156623223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls") pod "image-registry-8945c7fd4-h4fdq" (UID: "4637ce97-9134-469e-9b3c-388508a3858b") : secret "image-registry-tls" not found
Sep 09 15:20:54.934432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:54.934394    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:20:55.037048 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:55.037011    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:20:55.970484 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:55.970443    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:20:56.070198 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:56.070162    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:20:57.136812 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:57.136779    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:20:57.196832 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:57.196800    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:20:57.243738 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:57.243700    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:20:58.188024 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:58.187993    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:20:58.289170 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:58.289126    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:20:58.639734 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:58.639702    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:20:58.697331 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:58.697301    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:20:59.140088 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:59.140050    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:20:59.573835 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:59.573738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-62m26\" (UID: \"18973d3c-7fa4-4b71-ade6-1cd976413494\") " pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:20:59.573835 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:59.573827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:20:59.574256 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:20:59.573856    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:20:59.574256 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:59.573901    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 15:20:59.574256 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:59.573951    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:20:59.574256 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:59.574023    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 15:20:59.574256 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:59.574031    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert podName:18973d3c-7fa4-4b71-ade6-1cd976413494 nodeName:}" failed. No retries permitted until 2026-09-09 15:21:15.5739615 +0000 UTC m=+65.216003481 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-62m26" (UID: "18973d3c-7fa4-4b71-ade6-1cd976413494") : secret "networking-console-plugin-cert" not found
Sep 09 15:20:59.574256 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:59.574040    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8945c7fd4-h4fdq: secret "image-registry-tls" not found
Sep 09 15:20:59.574256 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:59.574052    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls podName:ed80dd24-f513-4215-9f41-c33847a78c80 nodeName:}" failed. No retries permitted until 2026-09-09 15:21:15.574044857 +0000 UTC m=+65.216086824 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls") pod "dns-default-rdv4c" (UID: "ed80dd24-f513-4215-9f41-c33847a78c80") : secret "dns-default-metrics-tls" not found
Sep 09 15:20:59.574256 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:20:59.574083    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls podName:4637ce97-9134-469e-9b3c-388508a3858b nodeName:}" failed. No retries permitted until 2026-09-09 15:21:15.574068728 +0000 UTC m=+65.216110698 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls") pod "image-registry-8945c7fd4-h4fdq" (UID: "4637ce97-9134-469e-9b3c-388508a3858b") : secret "image-registry-tls" not found
Sep 09 15:21:09.243811 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:09.243773    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:21:11.044191 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:11.044164    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-42r95"
Sep 09 15:21:11.189119 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:11.189081    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:21:11.566804 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:11.566759    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:21:14.757170 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:14.757131    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:21:15.593117 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:15.593080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-62m26\" (UID: \"18973d3c-7fa4-4b71-ade6-1cd976413494\") " pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:21:15.593293 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:15.593132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:21:15.593293 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:15.593157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:21:15.593293 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:21:15.593235    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 15:21:15.593293 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:21:15.593249    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 15:21:15.593293 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:21:15.593262    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:21:15.593293 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:21:15.593267    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8945c7fd4-h4fdq: secret "image-registry-tls" not found
Sep 09 15:21:15.593486 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:21:15.593306    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert podName:18973d3c-7fa4-4b71-ade6-1cd976413494 nodeName:}" failed. No retries permitted until 2026-09-09 15:21:47.593289806 +0000 UTC m=+97.235331793 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-62m26" (UID: "18973d3c-7fa4-4b71-ade6-1cd976413494") : secret "networking-console-plugin-cert" not found
Sep 09 15:21:15.593486 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:21:15.593321    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls podName:4637ce97-9134-469e-9b3c-388508a3858b nodeName:}" failed. No retries permitted until 2026-09-09 15:21:47.593315143 +0000 UTC m=+97.235357109 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls") pod "image-registry-8945c7fd4-h4fdq" (UID: "4637ce97-9134-469e-9b3c-388508a3858b") : secret "image-registry-tls" not found
Sep 09 15:21:15.593486 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:21:15.593331    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls podName:ed80dd24-f513-4215-9f41-c33847a78c80 nodeName:}" failed. No retries permitted until 2026-09-09 15:21:47.593326111 +0000 UTC m=+97.235368078 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls") pod "dns-default-rdv4c" (UID: "ed80dd24-f513-4215-9f41-c33847a78c80") : secret "dns-default-metrics-tls" not found
Sep 09 15:21:15.609987 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:15.609957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:21:15.711234 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:15.711203    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:21:16.273971 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:16.273935    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:21:16.324993 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:16.324960    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:21:16.499092 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:16.499050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs\") pod \"network-metrics-daemon-nhtcj\" (UID: \"7052f7a0-16d5-4115-82ae-34f07e9db9f9\") " pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:21:16.501653 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:16.501638    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 15:21:16.509409 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:21:16.509392    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 15:21:16.509457 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:21:16.509444    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs podName:7052f7a0-16d5-4115-82ae-34f07e9db9f9 nodeName:}" failed. No retries permitted until 2026-09-09 15:22:20.50942932 +0000 UTC m=+130.151471300 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs") pod "network-metrics-daemon-nhtcj" (UID: "7052f7a0-16d5-4115-82ae-34f07e9db9f9") : secret "metrics-daemon-secret" not found
Sep 09 15:21:16.600131 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:16.600064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret\") pod \"global-pull-secret-syncer-h4r6t\" (UID: \"560288f5-fbb4-4d6b-84bd-0987afe4c482\") " pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:21:16.602572 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:16.602551    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 15:21:16.613461 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:16.613432    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/560288f5-fbb4-4d6b-84bd-0987afe4c482-original-pull-secret\") pod \"global-pull-secret-syncer-h4r6t\" (UID: \"560288f5-fbb4-4d6b-84bd-0987afe4c482\") " pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:21:16.678595 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:16.678561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4r6t"
Sep 09 15:21:16.701151 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:16.701123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nms7\" (UniqueName: \"kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7\") pod \"network-check-target-7x79b\" (UID: \"b1406d46-d991-410d-bc6d-be16046a2fd5\") " pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:21:16.703994 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:16.703958    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 15:21:16.713741 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:16.713722    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 15:21:16.726065 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:16.726008    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nms7\" (UniqueName: \"kubernetes.io/projected/b1406d46-d991-410d-bc6d-be16046a2fd5-kube-api-access-6nms7\") pod \"network-check-target-7x79b\" (UID: \"b1406d46-d991-410d-bc6d-be16046a2fd5\") " pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:21:16.964840 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:16.964738    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h4r6t"]
Sep 09 15:21:16.967279 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:21:16.967250    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560288f5_fbb4_4d6b_84bd_0987afe4c482.slice/crio-08c0526b75c557ffe12cf4edce4686273efc132ae1aa6109d87fbfdb592cb9d6 WatchSource:0}: Error finding container 08c0526b75c557ffe12cf4edce4686273efc132ae1aa6109d87fbfdb592cb9d6: Status 404 returned error can't find the container with id 08c0526b75c557ffe12cf4edce4686273efc132ae1aa6109d87fbfdb592cb9d6
Sep 09 15:21:16.990592 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:16.990569    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7889l\""
Sep 09 15:21:16.998595 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:16.998578    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:21:17.136954 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:17.136912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h4r6t" event={"ID":"560288f5-fbb4-4d6b-84bd-0987afe4c482","Type":"ContainerStarted","Data":"08c0526b75c557ffe12cf4edce4686273efc132ae1aa6109d87fbfdb592cb9d6"}
Sep 09 15:21:17.262873 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:17.262805    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7x79b"]
Sep 09 15:21:17.265834 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:21:17.265796    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1406d46_d991_410d_bc6d_be16046a2fd5.slice/crio-6578ce31024566eef744ddf122057220749bfc73c174de67b10e3d18e8480214 WatchSource:0}: Error finding container 6578ce31024566eef744ddf122057220749bfc73c174de67b10e3d18e8480214: Status 404 returned error can't find the container with id 6578ce31024566eef744ddf122057220749bfc73c174de67b10e3d18e8480214
Sep 09 15:21:18.141293 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:18.141250    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7x79b" event={"ID":"b1406d46-d991-410d-bc6d-be16046a2fd5","Type":"ContainerStarted","Data":"6578ce31024566eef744ddf122057220749bfc73c174de67b10e3d18e8480214"}
Sep 09 15:21:21.151369 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:21.151327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7x79b" event={"ID":"b1406d46-d991-410d-bc6d-be16046a2fd5","Type":"ContainerStarted","Data":"53d81a923f3372890e749ac872cbda971e8a00117c8f1ad8675cfcc216460bb9"}
Sep 09 15:21:21.151767 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:21.151420    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:21:21.165030 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:21.164967    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7x79b" podStartSLOduration=67.115203018 podStartE2EDuration="1m10.164950891s" totalImagesPullingTime="3.049747873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:21:21.164677941 +0000 UTC m=+70.806719931" watchObservedRunningTime="2026-09-09 15:21:21.164950891 +0000 UTC m=+70.806992904"
Sep 09 15:21:25.164119 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:25.164084    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h4r6t" event={"ID":"560288f5-fbb4-4d6b-84bd-0987afe4c482","Type":"ContainerStarted","Data":"06bac1e8e9067c903c2818aa5619abeb77aaf92244125e5c7d20cf49d6fff465"}
Sep 09 15:21:25.178104 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:25.178061    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-h4r6t" podStartSLOduration=66.300618874 podStartE2EDuration="1m14.178047523s" totalImagesPullingTime="7.877428649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:21:25.177223604 +0000 UTC m=+74.819265591" watchObservedRunningTime="2026-09-09 15:21:25.178047523 +0000 UTC m=+74.820089511"
Sep 09 15:21:36.542917 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:36.542767    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:21:36.597700 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:36.597669    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:21:47.618213 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:47.618170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-62m26\" (UID: \"18973d3c-7fa4-4b71-ade6-1cd976413494\") " pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:21:47.618213 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:47.618220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:21:47.618627 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:47.618242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:21:47.618627 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:21:47.618325    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 15:21:47.618627 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:21:47.618343    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 15:21:47.618627 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:21:47.618363    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8945c7fd4-h4fdq: secret "image-registry-tls" not found
Sep 09 15:21:47.618627 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:21:47.618340    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:21:47.618627 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:21:47.618405    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert podName:18973d3c-7fa4-4b71-ade6-1cd976413494 nodeName:}" failed. No retries permitted until 2026-09-09 15:22:51.618390033 +0000 UTC m=+161.260432017 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-62m26" (UID: "18973d3c-7fa4-4b71-ade6-1cd976413494") : secret "networking-console-plugin-cert" not found
Sep 09 15:21:47.618627 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:21:47.618452    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls podName:4637ce97-9134-469e-9b3c-388508a3858b nodeName:}" failed. No retries permitted until 2026-09-09 15:22:51.618437789 +0000 UTC m=+161.260479762 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls") pod "image-registry-8945c7fd4-h4fdq" (UID: "4637ce97-9134-469e-9b3c-388508a3858b") : secret "image-registry-tls" not found
Sep 09 15:21:47.618627 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:21:47.618463    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls podName:ed80dd24-f513-4215-9f41-c33847a78c80 nodeName:}" failed. No retries permitted until 2026-09-09 15:22:51.618457572 +0000 UTC m=+161.260499539 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls") pod "dns-default-rdv4c" (UID: "ed80dd24-f513-4215-9f41-c33847a78c80") : secret "dns-default-metrics-tls" not found
Sep 09 15:21:52.158460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:21:52.158429    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7x79b"
Sep 09 15:22:20.548907 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:22:20.548857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs\") pod \"network-metrics-daemon-nhtcj\" (UID: \"7052f7a0-16d5-4115-82ae-34f07e9db9f9\") " pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:22:20.549502 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:22:20.549071    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 15:22:20.549502 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:22:20.549174    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs podName:7052f7a0-16d5-4115-82ae-34f07e9db9f9 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:22.549151564 +0000 UTC m=+252.191193536 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs") pod "network-metrics-daemon-nhtcj" (UID: "7052f7a0-16d5-4115-82ae-34f07e9db9f9") : secret "metrics-daemon-secret" not found
Sep 09 15:22:46.759405 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:22:46.759349    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-62m26" podUID="18973d3c-7fa4-4b71-ade6-1cd976413494"
Sep 09 15:22:46.767547 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:22:46.767515    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq" podUID="4637ce97-9134-469e-9b3c-388508a3858b"
Sep 09 15:22:46.774650 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:22:46.774625    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-rdv4c" podUID="ed80dd24-f513-4215-9f41-c33847a78c80"
Sep 09 15:22:47.352038 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:22:47.352010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:22:47.352038 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:22:47.352030    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:22:47.352238 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:22:47.352016    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:22:47.896724 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:22:47.896671    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:22:51.667587 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:22:51.667536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:22:51.667587 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:22:51.667591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:22:51.668116 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:22:51.667661    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-62m26\" (UID: \"18973d3c-7fa4-4b71-ade6-1cd976413494\") " pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:22:51.668116 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:22:51.667692    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 15:22:51.668116 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:22:51.667717    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8945c7fd4-h4fdq: secret "image-registry-tls" not found
Sep 09 15:22:51.668116 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:22:51.667751    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 15:22:51.668116 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:22:51.667766    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 15:22:51.668116 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:22:51.667788    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls podName:4637ce97-9134-469e-9b3c-388508a3858b nodeName:}" failed. No retries permitted until 2026-09-09 15:24:53.667770827 +0000 UTC m=+283.309812798 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls") pod "image-registry-8945c7fd4-h4fdq" (UID: "4637ce97-9134-469e-9b3c-388508a3858b") : secret "image-registry-tls" not found
Sep 09 15:22:51.668116 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:22:51.667825    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert podName:18973d3c-7fa4-4b71-ade6-1cd976413494 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:53.667807531 +0000 UTC m=+283.309849517 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-62m26" (UID: "18973d3c-7fa4-4b71-ade6-1cd976413494") : secret "networking-console-plugin-cert" not found
Sep 09 15:22:51.668116 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:22:51.667843    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls podName:ed80dd24-f513-4215-9f41-c33847a78c80 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:53.66783372 +0000 UTC m=+283.309875693 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls") pod "dns-default-rdv4c" (UID: "ed80dd24-f513-4215-9f41-c33847a78c80") : secret "dns-default-metrics-tls" not found
Sep 09 15:23:02.861440 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:23:02.861355    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:23:50.277723 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:23:50.277693    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:10.415285 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.415248    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"]
Sep 09 15:24:10.417058 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.417041    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n"]
Sep 09 15:24:10.417177 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.417158    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"
Sep 09 15:24:10.418813 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.418753    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-vsgld"]
Sep 09 15:24:10.418951 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.418930    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n"
Sep 09 15:24:10.419729 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.419710    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 15:24:10.420340 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.420321    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 15:24:10.420426 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.420413    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:10.421234 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.421217    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 15:24:10.421400 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.421380    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cgf4r\""
Sep 09 15:24:10.421500 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.421426    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 15:24:10.421500 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.421437    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 15:24:10.421500 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.421476    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 15:24:10.421662 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.421523    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-d97gh\""
Sep 09 15:24:10.421711 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.421684    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 15:24:10.422644 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.422628    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 15:24:10.423962 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.423941    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 15:24:10.424140 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.424039    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 15:24:10.424140 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.424076    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 15:24:10.424649 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.424530    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tgg62\""
Sep 09 15:24:10.436852 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.435122    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"]
Sep 09 15:24:10.437139 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.437116    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 15:24:10.437220 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.437207    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-vsgld"]
Sep 09 15:24:10.452531 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.452502    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n"]
Sep 09 15:24:10.505790 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.505758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f99z6\" (UniqueName: \"kubernetes.io/projected/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-kube-api-access-f99z6\") pod \"cluster-monitoring-operator-6cdcc96965-ntzdf\" (UID: \"3f4e6b8e-8887-4c02-bc23-4bdc6227850a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"
Sep 09 15:24:10.505790 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.505790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0231161f-516e-4130-a93e-4bb9a5e8fe49-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-psj4n\" (UID: \"0231161f-516e-4130-a93e-4bb9a5e8fe49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n"
Sep 09 15:24:10.505958 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.505810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9mj9\" (UniqueName: \"kubernetes.io/projected/0231161f-516e-4130-a93e-4bb9a5e8fe49-kube-api-access-j9mj9\") pod \"cluster-samples-operator-7cdcfcf76c-psj4n\" (UID: \"0231161f-516e-4130-a93e-4bb9a5e8fe49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n"
Sep 09 15:24:10.505958 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.505848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-ntzdf\" (UID: \"3f4e6b8e-8887-4c02-bc23-4bdc6227850a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"
Sep 09 15:24:10.505958 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.505863    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc2g2\" (UniqueName: \"kubernetes.io/projected/dc2e93bc-cc28-454c-9526-13e2837cdd14-kube-api-access-dc2g2\") pod \"console-operator-85b454874b-vsgld\" (UID: \"dc2e93bc-cc28-454c-9526-13e2837cdd14\") " pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:10.505958 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.505933    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc2e93bc-cc28-454c-9526-13e2837cdd14-trusted-ca\") pod \"console-operator-85b454874b-vsgld\" (UID: \"dc2e93bc-cc28-454c-9526-13e2837cdd14\") " pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:10.506121 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.506004    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc2e93bc-cc28-454c-9526-13e2837cdd14-config\") pod \"console-operator-85b454874b-vsgld\" (UID: \"dc2e93bc-cc28-454c-9526-13e2837cdd14\") " pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:10.506121 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.506023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc2e93bc-cc28-454c-9526-13e2837cdd14-serving-cert\") pod \"console-operator-85b454874b-vsgld\" (UID: \"dc2e93bc-cc28-454c-9526-13e2837cdd14\") " pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:10.506121 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.506060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ntzdf\" (UID: \"3f4e6b8e-8887-4c02-bc23-4bdc6227850a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"
Sep 09 15:24:10.514898 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.514867    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-llcq8"]
Sep 09 15:24:10.516652 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.516639    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-llcq8"
Sep 09 15:24:10.519316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.519291    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-54jtg\""
Sep 09 15:24:10.528612 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.528588    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-llcq8"]
Sep 09 15:24:10.606417 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.606380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ntzdf\" (UID: \"3f4e6b8e-8887-4c02-bc23-4bdc6227850a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"
Sep 09 15:24:10.606417 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.606420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-998ff\" (UniqueName: \"kubernetes.io/projected/dc58dad7-6393-4b38-91ab-d768357c1eb1-kube-api-access-998ff\") pod \"network-check-source-685456655f-llcq8\" (UID: \"dc58dad7-6393-4b38-91ab-d768357c1eb1\") " pod="openshift-network-diagnostics/network-check-source-685456655f-llcq8"
Sep 09 15:24:10.606645 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.606454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f99z6\" (UniqueName: \"kubernetes.io/projected/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-kube-api-access-f99z6\") pod \"cluster-monitoring-operator-6cdcc96965-ntzdf\" (UID: \"3f4e6b8e-8887-4c02-bc23-4bdc6227850a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"
Sep 09 15:24:10.606645 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.606497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0231161f-516e-4130-a93e-4bb9a5e8fe49-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-psj4n\" (UID: \"0231161f-516e-4130-a93e-4bb9a5e8fe49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n"
Sep 09 15:24:10.606645 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.606526    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9mj9\" (UniqueName: \"kubernetes.io/projected/0231161f-516e-4130-a93e-4bb9a5e8fe49-kube-api-access-j9mj9\") pod \"cluster-samples-operator-7cdcfcf76c-psj4n\" (UID: \"0231161f-516e-4130-a93e-4bb9a5e8fe49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n"
Sep 09 15:24:10.606645 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.606558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-ntzdf\" (UID: \"3f4e6b8e-8887-4c02-bc23-4bdc6227850a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"
Sep 09 15:24:10.606645 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:10.606570    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 15:24:10.606645 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:10.606591    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 15:24:10.606645 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.606575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc2g2\" (UniqueName: \"kubernetes.io/projected/dc2e93bc-cc28-454c-9526-13e2837cdd14-kube-api-access-dc2g2\") pod \"console-operator-85b454874b-vsgld\" (UID: \"dc2e93bc-cc28-454c-9526-13e2837cdd14\") " pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:10.606645 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:10.606649    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls podName:3f4e6b8e-8887-4c02-bc23-4bdc6227850a nodeName:}" failed. No retries permitted until 2026-09-09 15:24:11.106626731 +0000 UTC m=+240.748668710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-ntzdf" (UID: "3f4e6b8e-8887-4c02-bc23-4bdc6227850a") : secret "cluster-monitoring-operator-tls" not found
Sep 09 15:24:10.607064 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:10.606727    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0231161f-516e-4130-a93e-4bb9a5e8fe49-samples-operator-tls podName:0231161f-516e-4130-a93e-4bb9a5e8fe49 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:11.106709016 +0000 UTC m=+240.748750991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0231161f-516e-4130-a93e-4bb9a5e8fe49-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-psj4n" (UID: "0231161f-516e-4130-a93e-4bb9a5e8fe49") : secret "samples-operator-tls" not found
Sep 09 15:24:10.607064 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.606804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc2e93bc-cc28-454c-9526-13e2837cdd14-trusted-ca\") pod \"console-operator-85b454874b-vsgld\" (UID: \"dc2e93bc-cc28-454c-9526-13e2837cdd14\") " pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:10.607064 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.606851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc2e93bc-cc28-454c-9526-13e2837cdd14-config\") pod \"console-operator-85b454874b-vsgld\" (UID: \"dc2e93bc-cc28-454c-9526-13e2837cdd14\") " pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:10.607064 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.606875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc2e93bc-cc28-454c-9526-13e2837cdd14-serving-cert\") pod \"console-operator-85b454874b-vsgld\" (UID: \"dc2e93bc-cc28-454c-9526-13e2837cdd14\") " pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:10.607480 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.607455    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc2e93bc-cc28-454c-9526-13e2837cdd14-config\") pod \"console-operator-85b454874b-vsgld\" (UID: \"dc2e93bc-cc28-454c-9526-13e2837cdd14\") " pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:10.607574 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.607481    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-ntzdf\" (UID: \"3f4e6b8e-8887-4c02-bc23-4bdc6227850a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"
Sep 09 15:24:10.607679 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.607612    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc2e93bc-cc28-454c-9526-13e2837cdd14-trusted-ca\") pod \"console-operator-85b454874b-vsgld\" (UID: \"dc2e93bc-cc28-454c-9526-13e2837cdd14\") " pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:10.609287 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.609270    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc2e93bc-cc28-454c-9526-13e2837cdd14-serving-cert\") pod \"console-operator-85b454874b-vsgld\" (UID: \"dc2e93bc-cc28-454c-9526-13e2837cdd14\") " pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:10.619301 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.619272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9mj9\" (UniqueName: \"kubernetes.io/projected/0231161f-516e-4130-a93e-4bb9a5e8fe49-kube-api-access-j9mj9\") pod \"cluster-samples-operator-7cdcfcf76c-psj4n\" (UID: \"0231161f-516e-4130-a93e-4bb9a5e8fe49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n"
Sep 09 15:24:10.619395 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.619313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f99z6\" (UniqueName: \"kubernetes.io/projected/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-kube-api-access-f99z6\") pod \"cluster-monitoring-operator-6cdcc96965-ntzdf\" (UID: \"3f4e6b8e-8887-4c02-bc23-4bdc6227850a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"
Sep 09 15:24:10.619435 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.619415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc2g2\" (UniqueName: \"kubernetes.io/projected/dc2e93bc-cc28-454c-9526-13e2837cdd14-kube-api-access-dc2g2\") pod \"console-operator-85b454874b-vsgld\" (UID: \"dc2e93bc-cc28-454c-9526-13e2837cdd14\") " pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:10.648737 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.648712    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:10.707919 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.707868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-998ff\" (UniqueName: \"kubernetes.io/projected/dc58dad7-6393-4b38-91ab-d768357c1eb1-kube-api-access-998ff\") pod \"network-check-source-685456655f-llcq8\" (UID: \"dc58dad7-6393-4b38-91ab-d768357c1eb1\") " pod="openshift-network-diagnostics/network-check-source-685456655f-llcq8"
Sep 09 15:24:10.715733 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.715713    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-998ff\" (UniqueName: \"kubernetes.io/projected/dc58dad7-6393-4b38-91ab-d768357c1eb1-kube-api-access-998ff\") pod \"network-check-source-685456655f-llcq8\" (UID: \"dc58dad7-6393-4b38-91ab-d768357c1eb1\") " pod="openshift-network-diagnostics/network-check-source-685456655f-llcq8"
Sep 09 15:24:10.724740 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.724712    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:10.742678 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.742656    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tgg62\""
Sep 09 15:24:10.750735 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.750719    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:10.826683 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.826655    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-54jtg\""
Sep 09 15:24:10.834585 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:10.834559    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-llcq8"
Sep 09 15:24:11.015825 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:11.015748    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-vsgld"]
Sep 09 15:24:11.018366 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:24:11.018342    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2e93bc_cc28_454c_9526_13e2837cdd14.slice/crio-5c5fecc763618ce61850c4d73d40b97ed3a800a1ec26386d03fd64f7fc529871 WatchSource:0}: Error finding container 5c5fecc763618ce61850c4d73d40b97ed3a800a1ec26386d03fd64f7fc529871: Status 404 returned error can't find the container with id 5c5fecc763618ce61850c4d73d40b97ed3a800a1ec26386d03fd64f7fc529871
Sep 09 15:24:11.097559 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:11.097524    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-llcq8"]
Sep 09 15:24:11.100122 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:24:11.100097    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc58dad7_6393_4b38_91ab_d768357c1eb1.slice/crio-6b78ef0a92051e874a4adb096660f797bd00870b145ddb2100a815919d10f21d WatchSource:0}: Error finding container 6b78ef0a92051e874a4adb096660f797bd00870b145ddb2100a815919d10f21d: Status 404 returned error can't find the container with id 6b78ef0a92051e874a4adb096660f797bd00870b145ddb2100a815919d10f21d
Sep 09 15:24:11.111226 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:11.111204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0231161f-516e-4130-a93e-4bb9a5e8fe49-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-psj4n\" (UID: \"0231161f-516e-4130-a93e-4bb9a5e8fe49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n"
Sep 09 15:24:11.111351 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:11.111332    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ntzdf\" (UID: \"3f4e6b8e-8887-4c02-bc23-4bdc6227850a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"
Sep 09 15:24:11.111407 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:11.111341    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 15:24:11.111458 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:11.111376    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 15:24:11.111493 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:11.111458    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0231161f-516e-4130-a93e-4bb9a5e8fe49-samples-operator-tls podName:0231161f-516e-4130-a93e-4bb9a5e8fe49 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:12.111439088 +0000 UTC m=+241.753481072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0231161f-516e-4130-a93e-4bb9a5e8fe49-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-psj4n" (UID: "0231161f-516e-4130-a93e-4bb9a5e8fe49") : secret "samples-operator-tls" not found
Sep 09 15:24:11.111538 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:11.111492    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls podName:3f4e6b8e-8887-4c02-bc23-4bdc6227850a nodeName:}" failed. No retries permitted until 2026-09-09 15:24:12.111479503 +0000 UTC m=+241.753521483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-ntzdf" (UID: "3f4e6b8e-8887-4c02-bc23-4bdc6227850a") : secret "cluster-monitoring-operator-tls" not found
Sep 09 15:24:11.550831 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:11.550797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-llcq8" event={"ID":"dc58dad7-6393-4b38-91ab-d768357c1eb1","Type":"ContainerStarted","Data":"2f66a60ee82980a702f8cb59bb1567fbe96dda230c8ed39aaea73421f31db931"}
Sep 09 15:24:11.551297 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:11.550862    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-llcq8" event={"ID":"dc58dad7-6393-4b38-91ab-d768357c1eb1","Type":"ContainerStarted","Data":"6b78ef0a92051e874a4adb096660f797bd00870b145ddb2100a815919d10f21d"}
Sep 09 15:24:11.551963 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:11.551940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-vsgld" event={"ID":"dc2e93bc-cc28-454c-9526-13e2837cdd14","Type":"ContainerStarted","Data":"5c5fecc763618ce61850c4d73d40b97ed3a800a1ec26386d03fd64f7fc529871"}
Sep 09 15:24:11.564948 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:11.564907    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-llcq8" podStartSLOduration=1.564891542 podStartE2EDuration="1.564891542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:24:11.564284376 +0000 UTC m=+241.206326368" watchObservedRunningTime="2026-09-09 15:24:11.564891542 +0000 UTC m=+241.206933553"
Sep 09 15:24:12.121090 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:12.121043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ntzdf\" (UID: \"3f4e6b8e-8887-4c02-bc23-4bdc6227850a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"
Sep 09 15:24:12.121283 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:12.121161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0231161f-516e-4130-a93e-4bb9a5e8fe49-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-psj4n\" (UID: \"0231161f-516e-4130-a93e-4bb9a5e8fe49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n"
Sep 09 15:24:12.121352 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:12.121305    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 15:24:12.121406 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:12.121354    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 15:24:12.121406 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:12.121394    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls podName:3f4e6b8e-8887-4c02-bc23-4bdc6227850a nodeName:}" failed. No retries permitted until 2026-09-09 15:24:14.121370957 +0000 UTC m=+243.763412939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-ntzdf" (UID: "3f4e6b8e-8887-4c02-bc23-4bdc6227850a") : secret "cluster-monitoring-operator-tls" not found
Sep 09 15:24:12.121513 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:12.121421    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0231161f-516e-4130-a93e-4bb9a5e8fe49-samples-operator-tls podName:0231161f-516e-4130-a93e-4bb9a5e8fe49 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:14.121404476 +0000 UTC m=+243.763446461 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0231161f-516e-4130-a93e-4bb9a5e8fe49-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-psj4n" (UID: "0231161f-516e-4130-a93e-4bb9a5e8fe49") : secret "samples-operator-tls" not found
Sep 09 15:24:13.085584 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:13.085528    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:14.139156 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:14.139112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ntzdf\" (UID: \"3f4e6b8e-8887-4c02-bc23-4bdc6227850a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"
Sep 09 15:24:14.139558 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:14.139184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0231161f-516e-4130-a93e-4bb9a5e8fe49-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-psj4n\" (UID: \"0231161f-516e-4130-a93e-4bb9a5e8fe49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n"
Sep 09 15:24:14.139558 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:14.139257    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 15:24:14.139558 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:14.139311    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 15:24:14.139558 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:14.139323    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls podName:3f4e6b8e-8887-4c02-bc23-4bdc6227850a nodeName:}" failed. No retries permitted until 2026-09-09 15:24:18.139307367 +0000 UTC m=+247.781349350 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-ntzdf" (UID: "3f4e6b8e-8887-4c02-bc23-4bdc6227850a") : secret "cluster-monitoring-operator-tls" not found
Sep 09 15:24:14.139558 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:14.139362    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0231161f-516e-4130-a93e-4bb9a5e8fe49-samples-operator-tls podName:0231161f-516e-4130-a93e-4bb9a5e8fe49 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:18.139347198 +0000 UTC m=+247.781389165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0231161f-516e-4130-a93e-4bb9a5e8fe49-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-psj4n" (UID: "0231161f-516e-4130-a93e-4bb9a5e8fe49") : secret "samples-operator-tls" not found
Sep 09 15:24:14.563969 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:14.563897    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/0.log"
Sep 09 15:24:14.563969 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:14.563936    2578 generic.go:356] "Generic (PLEG): container finished" podID="dc2e93bc-cc28-454c-9526-13e2837cdd14" containerID="3bab5487548ca3e112c7674651403d3506bad7f06670acfeb426d645205eefc4" exitCode=255
Sep 09 15:24:14.564152 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:14.563999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-vsgld" event={"ID":"dc2e93bc-cc28-454c-9526-13e2837cdd14","Type":"ContainerDied","Data":"3bab5487548ca3e112c7674651403d3506bad7f06670acfeb426d645205eefc4"}
Sep 09 15:24:14.564232 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:14.564217    2578 scope.go:117] "RemoveContainer" containerID="3bab5487548ca3e112c7674651403d3506bad7f06670acfeb426d645205eefc4"
Sep 09 15:24:15.568500 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.568474    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/1.log"
Sep 09 15:24:15.568911 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.568873    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/0.log"
Sep 09 15:24:15.568960 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.568910    2578 generic.go:356] "Generic (PLEG): container finished" podID="dc2e93bc-cc28-454c-9526-13e2837cdd14" containerID="e748719af6e2679250769825d05688e95c875c7b4be4e18659c4738d57068bed" exitCode=255
Sep 09 15:24:15.569017 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.568959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-vsgld" event={"ID":"dc2e93bc-cc28-454c-9526-13e2837cdd14","Type":"ContainerDied","Data":"e748719af6e2679250769825d05688e95c875c7b4be4e18659c4738d57068bed"}
Sep 09 15:24:15.569070 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.569047    2578 scope.go:117] "RemoveContainer" containerID="3bab5487548ca3e112c7674651403d3506bad7f06670acfeb426d645205eefc4"
Sep 09 15:24:15.569263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.569238    2578 scope.go:117] "RemoveContainer" containerID="e748719af6e2679250769825d05688e95c875c7b4be4e18659c4738d57068bed"
Sep 09 15:24:15.569518 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:15.569432    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-vsgld_openshift-console-operator(dc2e93bc-cc28-454c-9526-13e2837cdd14)\"" pod="openshift-console-operator/console-operator-85b454874b-vsgld" podUID="dc2e93bc-cc28-454c-9526-13e2837cdd14"
Sep 09 15:24:15.626060 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.626025    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xnkml"]
Sep 09 15:24:15.628639 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.628622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:15.630724 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.630699    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 15:24:15.631386 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.631358    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 15:24:15.631490 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.631458    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 15:24:15.631546 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.631530    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 15:24:15.635131 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.631913    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6tbnq\""
Sep 09 15:24:15.640722 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.640700    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xnkml"]
Sep 09 15:24:15.641814 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.641793    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:15.702951 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.702919    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:15.753005 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.752959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:15.753136 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.753046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85da71af-045e-45d6-adc6-dca483128c4f-crio-socket\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:15.753177 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.753148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25nhj\" (UniqueName: \"kubernetes.io/projected/85da71af-045e-45d6-adc6-dca483128c4f-kube-api-access-25nhj\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:15.753215 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.753178    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85da71af-045e-45d6-adc6-dca483128c4f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:15.753215 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.753199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85da71af-045e-45d6-adc6-dca483128c4f-data-volume\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:15.771077 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.771044    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:15.793524 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.793500    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-r6c44"]
Sep 09 15:24:15.795702 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.795689    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-r6c44"
Sep 09 15:24:15.798179 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.798162    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 09 15:24:15.798297 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.798278    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-p2k69\""
Sep 09 15:24:15.798517 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.798504    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 09 15:24:15.806406 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.806387    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-r6c44"]
Sep 09 15:24:15.854316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.854257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85da71af-045e-45d6-adc6-dca483128c4f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:15.854316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.854291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85da71af-045e-45d6-adc6-dca483128c4f-data-volume\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:15.854488 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.854327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:15.854488 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:15.854381    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 15:24:15.854488 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:15.854404    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:15.854488 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:15.854448    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85da71af-045e-45d6-adc6-dca483128c4f-kube-rbac-proxy-cm podName:85da71af-045e-45d6-adc6-dca483128c4f nodeName:}" failed. No retries permitted until 2026-09-09 15:24:16.354432836 +0000 UTC m=+245.996474803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85da71af-045e-45d6-adc6-dca483128c4f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xnkml" (UID: "85da71af-045e-45d6-adc6-dca483128c4f") : configmap "kube-rbac-proxy" not found
Sep 09 15:24:15.854700 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.854497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85da71af-045e-45d6-adc6-dca483128c4f-crio-socket\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:15.854700 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:15.854508    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls podName:85da71af-045e-45d6-adc6-dca483128c4f nodeName:}" failed. No retries permitted until 2026-09-09 15:24:16.354492789 +0000 UTC m=+245.996534770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xnkml" (UID: "85da71af-045e-45d6-adc6-dca483128c4f") : secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:15.854700 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.854552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85da71af-045e-45d6-adc6-dca483128c4f-crio-socket\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:15.854700 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.854564    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85da71af-045e-45d6-adc6-dca483128c4f-data-volume\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:15.854700 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.854593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25nhj\" (UniqueName: \"kubernetes.io/projected/85da71af-045e-45d6-adc6-dca483128c4f-kube-api-access-25nhj\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:15.862874 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.862857    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25nhj\" (UniqueName: \"kubernetes.io/projected/85da71af-045e-45d6-adc6-dca483128c4f-kube-api-access-25nhj\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:15.870283 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.870262    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:15.955583 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.955552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v4w8\" (UniqueName: \"kubernetes.io/projected/5c4d7ec2-3aa1-4a1b-ba22-83028a88db06-kube-api-access-4v4w8\") pod \"migrator-fdbb8fdff-r6c44\" (UID: \"5c4d7ec2-3aa1-4a1b-ba22-83028a88db06\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-r6c44"
Sep 09 15:24:15.972702 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:15.972676    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:16.056156 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:16.056125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v4w8\" (UniqueName: \"kubernetes.io/projected/5c4d7ec2-3aa1-4a1b-ba22-83028a88db06-kube-api-access-4v4w8\") pod \"migrator-fdbb8fdff-r6c44\" (UID: \"5c4d7ec2-3aa1-4a1b-ba22-83028a88db06\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-r6c44"
Sep 09 15:24:16.063606 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:16.063583    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v4w8\" (UniqueName: \"kubernetes.io/projected/5c4d7ec2-3aa1-4a1b-ba22-83028a88db06-kube-api-access-4v4w8\") pod \"migrator-fdbb8fdff-r6c44\" (UID: \"5c4d7ec2-3aa1-4a1b-ba22-83028a88db06\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-r6c44"
Sep 09 15:24:16.070345 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:16.070321    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:16.104426 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:16.104373    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-r6c44"
Sep 09 15:24:16.358694 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:16.358628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:16.358824 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:16.358751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85da71af-045e-45d6-adc6-dca483128c4f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:16.358824 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:16.358781    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:16.358903 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:16.358829    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 15:24:16.358903 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:16.358858    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls podName:85da71af-045e-45d6-adc6-dca483128c4f nodeName:}" failed. No retries permitted until 2026-09-09 15:24:17.358832635 +0000 UTC m=+247.000874621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xnkml" (UID: "85da71af-045e-45d6-adc6-dca483128c4f") : secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:16.358903 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:16.358876    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85da71af-045e-45d6-adc6-dca483128c4f-kube-rbac-proxy-cm podName:85da71af-045e-45d6-adc6-dca483128c4f nodeName:}" failed. No retries permitted until 2026-09-09 15:24:17.358865529 +0000 UTC m=+247.000907520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85da71af-045e-45d6-adc6-dca483128c4f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xnkml" (UID: "85da71af-045e-45d6-adc6-dca483128c4f") : configmap "kube-rbac-proxy" not found
Sep 09 15:24:16.368044 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:16.368018    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-r6c44"]
Sep 09 15:24:16.371048 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:24:16.371027    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4d7ec2_3aa1_4a1b_ba22_83028a88db06.slice/crio-4c387ce44f8a7712ae153dda950a6f72a88c6ab8d78087238b16d78020068561 WatchSource:0}: Error finding container 4c387ce44f8a7712ae153dda950a6f72a88c6ab8d78087238b16d78020068561: Status 404 returned error can't find the container with id 4c387ce44f8a7712ae153dda950a6f72a88c6ab8d78087238b16d78020068561
Sep 09 15:24:16.574756 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:16.574728    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/1.log"
Sep 09 15:24:16.575165 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:16.575107    2578 scope.go:117] "RemoveContainer" containerID="e748719af6e2679250769825d05688e95c875c7b4be4e18659c4738d57068bed"
Sep 09 15:24:16.575348 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:16.575328    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-vsgld_openshift-console-operator(dc2e93bc-cc28-454c-9526-13e2837cdd14)\"" pod="openshift-console-operator/console-operator-85b454874b-vsgld" podUID="dc2e93bc-cc28-454c-9526-13e2837cdd14"
Sep 09 15:24:16.575920 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:16.575900    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-r6c44" event={"ID":"5c4d7ec2-3aa1-4a1b-ba22-83028a88db06","Type":"ContainerStarted","Data":"4c387ce44f8a7712ae153dda950a6f72a88c6ab8d78087238b16d78020068561"}
Sep 09 15:24:17.026746 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:17.026682    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:17.127620 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:17.127553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:17.367317 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:17.367239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85da71af-045e-45d6-adc6-dca483128c4f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:17.367317 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:17.367298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:17.367530 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:17.367395    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 15:24:17.367530 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:17.367424    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:17.367530 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:17.367477    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85da71af-045e-45d6-adc6-dca483128c4f-kube-rbac-proxy-cm podName:85da71af-045e-45d6-adc6-dca483128c4f nodeName:}" failed. No retries permitted until 2026-09-09 15:24:19.367454044 +0000 UTC m=+249.009496023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85da71af-045e-45d6-adc6-dca483128c4f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xnkml" (UID: "85da71af-045e-45d6-adc6-dca483128c4f") : configmap "kube-rbac-proxy" not found
Sep 09 15:24:17.367530 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:17.367498    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls podName:85da71af-045e-45d6-adc6-dca483128c4f nodeName:}" failed. No retries permitted until 2026-09-09 15:24:19.367487936 +0000 UTC m=+249.009529904 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xnkml" (UID: "85da71af-045e-45d6-adc6-dca483128c4f") : secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:17.598966 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:17.598942    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v2hbm_9fc01e4b-4a6f-47f0-8220-ca62e74d7250/dns-node-resolver/0.log"
Sep 09 15:24:18.174822 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:18.174789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0231161f-516e-4130-a93e-4bb9a5e8fe49-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-psj4n\" (UID: \"0231161f-516e-4130-a93e-4bb9a5e8fe49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n"
Sep 09 15:24:18.175046 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:18.174913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ntzdf\" (UID: \"3f4e6b8e-8887-4c02-bc23-4bdc6227850a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"
Sep 09 15:24:18.175046 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:18.174970    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 15:24:18.175175 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:18.175051    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 15:24:18.175175 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:18.175062    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0231161f-516e-4130-a93e-4bb9a5e8fe49-samples-operator-tls podName:0231161f-516e-4130-a93e-4bb9a5e8fe49 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:26.175042365 +0000 UTC m=+255.817084332 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0231161f-516e-4130-a93e-4bb9a5e8fe49-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-psj4n" (UID: "0231161f-516e-4130-a93e-4bb9a5e8fe49") : secret "samples-operator-tls" not found
Sep 09 15:24:18.175175 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:18.175094    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls podName:3f4e6b8e-8887-4c02-bc23-4bdc6227850a nodeName:}" failed. No retries permitted until 2026-09-09 15:24:26.175081848 +0000 UTC m=+255.817123815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-ntzdf" (UID: "3f4e6b8e-8887-4c02-bc23-4bdc6227850a") : secret "cluster-monitoring-operator-tls" not found
Sep 09 15:24:18.398575 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:18.398544    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qn7pt_0b0bf69a-0e36-446f-8748-a84c778fe36e/node-ca/0.log"
Sep 09 15:24:18.414313 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:18.414282    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:18.470918 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:18.470848    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:18.584606 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:18.584574    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-r6c44" event={"ID":"5c4d7ec2-3aa1-4a1b-ba22-83028a88db06","Type":"ContainerStarted","Data":"58cc8c3a0db4c93dedc6521ec2d0e5b4a5378917d4db1019e9c08e186a0315b9"}
Sep 09 15:24:18.584724 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:18.584625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-r6c44" event={"ID":"5c4d7ec2-3aa1-4a1b-ba22-83028a88db06","Type":"ContainerStarted","Data":"c754d5aabccc1e0903cafff559a7e4505756c3404f41abdd3a5552896efcd3bf"}
Sep 09 15:24:18.599417 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:18.599373    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-r6c44" podStartSLOduration=2.423899151 podStartE2EDuration="3.599358678s" totalImagesPullingTime="1.175459527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:24:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:24:18.599155149 +0000 UTC m=+248.241197138" watchObservedRunningTime="2026-09-09 15:24:18.599358678 +0000 UTC m=+248.241400668"
Sep 09 15:24:18.673721 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:18.673695    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:18.727059 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:18.726995    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:19.209867 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:19.209817    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:19.384472 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:19.384435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85da71af-045e-45d6-adc6-dca483128c4f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:19.384639 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:19.384489    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:19.384639 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:19.384563    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 15:24:19.384639 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:19.384590    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:19.384639 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:19.384631    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85da71af-045e-45d6-adc6-dca483128c4f-kube-rbac-proxy-cm podName:85da71af-045e-45d6-adc6-dca483128c4f nodeName:}" failed. No retries permitted until 2026-09-09 15:24:23.384616082 +0000 UTC m=+253.026658049 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85da71af-045e-45d6-adc6-dca483128c4f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xnkml" (UID: "85da71af-045e-45d6-adc6-dca483128c4f") : configmap "kube-rbac-proxy" not found
Sep 09 15:24:19.384775 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:19.384647    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls podName:85da71af-045e-45d6-adc6-dca483128c4f nodeName:}" failed. No retries permitted until 2026-09-09 15:24:23.384640818 +0000 UTC m=+253.026682785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xnkml" (UID: "85da71af-045e-45d6-adc6-dca483128c4f") : secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:19.599288 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:19.599201    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j58rw_7b5c31a0-d5d8-4e50-ad6b-e0697ec8c073/serve-healthcheck-canary/0.log"
Sep 09 15:24:20.751255 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:20.751219    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:20.751635 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:20.751274    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:20.751635 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:20.751576    2578 scope.go:117] "RemoveContainer" containerID="e748719af6e2679250769825d05688e95c875c7b4be4e18659c4738d57068bed"
Sep 09 15:24:20.751731 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:20.751713    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-vsgld_openshift-console-operator(dc2e93bc-cc28-454c-9526-13e2837cdd14)\"" pod="openshift-console-operator/console-operator-85b454874b-vsgld" podUID="dc2e93bc-cc28-454c-9526-13e2837cdd14"
Sep 09 15:24:21.803957 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:21.803923    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:22.198820 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:22.198783    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:22.404741 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:22.404703    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:22.507638 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:22.507562    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:22.564080 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:22.564046    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:22.605572 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:22.605542    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs\") pod \"network-metrics-daemon-nhtcj\" (UID: \"7052f7a0-16d5-4115-82ae-34f07e9db9f9\") " pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:24:22.605714 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:22.605691    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 15:24:22.605754 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:22.605748    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs podName:7052f7a0-16d5-4115-82ae-34f07e9db9f9 nodeName:}" failed. No retries permitted until 2026-09-09 15:26:24.605733559 +0000 UTC m=+374.247775526 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs") pod "network-metrics-daemon-nhtcj" (UID: "7052f7a0-16d5-4115-82ae-34f07e9db9f9") : secret "metrics-daemon-secret" not found
Sep 09 15:24:23.395858 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:23.395805    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:23.412094 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:23.412059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85da71af-045e-45d6-adc6-dca483128c4f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:23.412227 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:23.412129    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:23.412227 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:23.412185    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 15:24:23.412331 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:23.412240    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:23.412331 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:23.412254    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85da71af-045e-45d6-adc6-dca483128c4f-kube-rbac-proxy-cm podName:85da71af-045e-45d6-adc6-dca483128c4f nodeName:}" failed. No retries permitted until 2026-09-09 15:24:31.412235898 +0000 UTC m=+261.054277882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85da71af-045e-45d6-adc6-dca483128c4f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xnkml" (UID: "85da71af-045e-45d6-adc6-dca483128c4f") : configmap "kube-rbac-proxy" not found
Sep 09 15:24:23.412331 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:23.412294    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls podName:85da71af-045e-45d6-adc6-dca483128c4f nodeName:}" failed. No retries permitted until 2026-09-09 15:24:31.412277474 +0000 UTC m=+261.054319446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xnkml" (UID: "85da71af-045e-45d6-adc6-dca483128c4f") : secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:23.495244 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:23.495218    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:24.719849 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:24.719811    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:25.332817 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:25.332785    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:25.384499 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:25.384467    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:25.869338 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:25.869294    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:25.921285 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:25.921256    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:26.077580 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:26.077544    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:26.133168 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:26.133144    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:26.235364 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:26.235314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ntzdf\" (UID: \"3f4e6b8e-8887-4c02-bc23-4bdc6227850a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"
Sep 09 15:24:26.235515 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:26.235379    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0231161f-516e-4130-a93e-4bb9a5e8fe49-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-psj4n\" (UID: \"0231161f-516e-4130-a93e-4bb9a5e8fe49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n"
Sep 09 15:24:26.235515 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:26.235474    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 15:24:26.235610 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:26.235532    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls podName:3f4e6b8e-8887-4c02-bc23-4bdc6227850a nodeName:}" failed. No retries permitted until 2026-09-09 15:24:42.235518047 +0000 UTC m=+271.877560026 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-ntzdf" (UID: "3f4e6b8e-8887-4c02-bc23-4bdc6227850a") : secret "cluster-monitoring-operator-tls" not found
Sep 09 15:24:26.237792 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:26.237771    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0231161f-516e-4130-a93e-4bb9a5e8fe49-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-psj4n\" (UID: \"0231161f-516e-4130-a93e-4bb9a5e8fe49\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n"
Sep 09 15:24:26.244518 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:26.244496    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:26.336762 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:26.336738    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-d97gh\""
Sep 09 15:24:26.344630 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:26.344614    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n"
Sep 09 15:24:26.489314 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:26.489282    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:26.612962 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:26.612934    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n"]
Sep 09 15:24:26.687416 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:26.687386    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:27.092239 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:27.092157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:27.610936 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:27.610904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n" event={"ID":"0231161f-516e-4130-a93e-4bb9a5e8fe49","Type":"ContainerStarted","Data":"1011fb27fa7731048a88cd410093cce393d7a80426f6f8f3df6f460eb626b1eb"}
Sep 09 15:24:27.890099 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:27.890067    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:28.289532 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:28.289467    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:28.567460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:28.567378    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:28.615665 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:28.615632    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n" event={"ID":"0231161f-516e-4130-a93e-4bb9a5e8fe49","Type":"ContainerStarted","Data":"7f9fbc17daf2c2aebc7ebf8e624536245ca2d3d339edcf98be6b4cc328d3235d"}
Sep 09 15:24:28.615851 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:28.615699    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n" event={"ID":"0231161f-516e-4130-a93e-4bb9a5e8fe49","Type":"ContainerStarted","Data":"0d9f801442dc9fe5fded416da83c3cb956f695d89f6b42f930c10c170ca81542"}
Sep 09 15:24:28.631263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:28.631217    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-psj4n" podStartSLOduration=16.979041763 podStartE2EDuration="18.631202954s" totalImagesPullingTime="1.652161191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:24:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:24:28.629547688 +0000 UTC m=+258.271589677" watchObservedRunningTime="2026-09-09 15:24:28.631202954 +0000 UTC m=+258.273244943"
Sep 09 15:24:28.687249 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:28.687222    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:28.787857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:28.787811    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:28.888709 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:28.888681    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:28.944861 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:28.944800    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:29.909164 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:29.909134    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:30.357942 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:30.357824    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:30.482708 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:30.482673    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:30.688880 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:30.688825    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:30.884695 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:30.884627    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:31.480907 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:31.480872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85da71af-045e-45d6-adc6-dca483128c4f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:31.481347 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:31.480928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:31.481347 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:31.481042    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:31.481347 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:31.481101    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls podName:85da71af-045e-45d6-adc6-dca483128c4f nodeName:}" failed. No retries permitted until 2026-09-09 15:24:47.481086072 +0000 UTC m=+277.123128058 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xnkml" (UID: "85da71af-045e-45d6-adc6-dca483128c4f") : secret "insights-runtime-extractor-tls" not found
Sep 09 15:24:31.481457 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:31.481413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85da71af-045e-45d6-adc6-dca483128c4f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:31.899881 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:31.899852    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:32.361681 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:32.361607    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:32.482213 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:32.482180    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:32.861088 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:32.861057    2578 scope.go:117] "RemoveContainer" containerID="e748719af6e2679250769825d05688e95c875c7b4be4e18659c4738d57068bed"
Sep 09 15:24:33.631670 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:33.631638    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 15:24:33.632064 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:33.632043    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/1.log"
Sep 09 15:24:33.632111 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:33.632076    2578 generic.go:356] "Generic (PLEG): container finished" podID="dc2e93bc-cc28-454c-9526-13e2837cdd14" containerID="c7474639070de4ae23a6a00efdfc7687aedc2a5a8630e83020b40ba42301c3c6" exitCode=255
Sep 09 15:24:33.632142 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:33.632125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-vsgld" event={"ID":"dc2e93bc-cc28-454c-9526-13e2837cdd14","Type":"ContainerDied","Data":"c7474639070de4ae23a6a00efdfc7687aedc2a5a8630e83020b40ba42301c3c6"}
Sep 09 15:24:33.632191 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:33.632177    2578 scope.go:117] "RemoveContainer" containerID="e748719af6e2679250769825d05688e95c875c7b4be4e18659c4738d57068bed"
Sep 09 15:24:33.632486 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:33.632470    2578 scope.go:117] "RemoveContainer" containerID="c7474639070de4ae23a6a00efdfc7687aedc2a5a8630e83020b40ba42301c3c6"
Sep 09 15:24:33.632708 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:33.632680    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-vsgld_openshift-console-operator(dc2e93bc-cc28-454c-9526-13e2837cdd14)\"" pod="openshift-console-operator/console-operator-85b454874b-vsgld" podUID="dc2e93bc-cc28-454c-9526-13e2837cdd14"
Sep 09 15:24:34.637246 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:34.637218    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 15:24:35.757285 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:35.757252    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:37.405990 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:37.405937    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:37.509949 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:37.509920    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:39.607693 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:39.607660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:40.751301 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:40.751258    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:40.751301 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:40.751311    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:40.751732 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:40.751588    2578 scope.go:117] "RemoveContainer" containerID="c7474639070de4ae23a6a00efdfc7687aedc2a5a8630e83020b40ba42301c3c6"
Sep 09 15:24:40.751771 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:40.751732    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-vsgld_openshift-console-operator(dc2e93bc-cc28-454c-9526-13e2837cdd14)\"" pod="openshift-console-operator/console-operator-85b454874b-vsgld" podUID="dc2e93bc-cc28-454c-9526-13e2837cdd14"
Sep 09 15:24:40.973966 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:40.973935    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:41.074309 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:41.074239    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:42.264863 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:42.264829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ntzdf\" (UID: \"3f4e6b8e-8887-4c02-bc23-4bdc6227850a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"
Sep 09 15:24:42.267360 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:42.267331    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f4e6b8e-8887-4c02-bc23-4bdc6227850a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-ntzdf\" (UID: \"3f4e6b8e-8887-4c02-bc23-4bdc6227850a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"
Sep 09 15:24:42.530639 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:42.530565    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cgf4r\""
Sep 09 15:24:42.538530 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:42.538513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"
Sep 09 15:24:42.804309 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:42.804240    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf"]
Sep 09 15:24:42.806900 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:24:42.806870    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4e6b8e_8887_4c02_bc23_4bdc6227850a.slice/crio-eab3f85f9f0a6a5af3faf648f19b65caea9173a94bb9a760167652e3d90bb408 WatchSource:0}: Error finding container eab3f85f9f0a6a5af3faf648f19b65caea9173a94bb9a760167652e3d90bb408: Status 404 returned error can't find the container with id eab3f85f9f0a6a5af3faf648f19b65caea9173a94bb9a760167652e3d90bb408
Sep 09 15:24:43.666162 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:43.666118    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf" event={"ID":"3f4e6b8e-8887-4c02-bc23-4bdc6227850a","Type":"ContainerStarted","Data":"eab3f85f9f0a6a5af3faf648f19b65caea9173a94bb9a760167652e3d90bb408"}
Sep 09 15:24:44.967182 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:44.967096    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:45.070420 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:45.070390    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:45.126690 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:45.126656    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-4zzgj"]
Sep 09 15:24:45.129724 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:45.129701    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-4zzgj"
Sep 09 15:24:45.132087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:45.132063    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 09 15:24:45.132190 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:45.132114    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-456v7\""
Sep 09 15:24:45.136783 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:45.136760    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-4zzgj"]
Sep 09 15:24:45.142418 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:45.142400    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:45.189248 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:45.189216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e6e7e9e2-4a62-47f9-a603-1c77e465e43a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-4zzgj\" (UID: \"e6e7e9e2-4a62-47f9-a603-1c77e465e43a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-4zzgj"
Sep 09 15:24:45.290346 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:45.290270    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e6e7e9e2-4a62-47f9-a603-1c77e465e43a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-4zzgj\" (UID: \"e6e7e9e2-4a62-47f9-a603-1c77e465e43a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-4zzgj"
Sep 09 15:24:45.290447 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:45.290414    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 09 15:24:45.290489 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:45.290477    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6e7e9e2-4a62-47f9-a603-1c77e465e43a-tls-certificates podName:e6e7e9e2-4a62-47f9-a603-1c77e465e43a nodeName:}" failed. No retries permitted until 2026-09-09 15:24:45.790460619 +0000 UTC m=+275.432502613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/e6e7e9e2-4a62-47f9-a603-1c77e465e43a-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-4zzgj" (UID: "e6e7e9e2-4a62-47f9-a603-1c77e465e43a") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 09 15:24:45.674190 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:45.674154    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf" event={"ID":"3f4e6b8e-8887-4c02-bc23-4bdc6227850a","Type":"ContainerStarted","Data":"5f240f11c5ce8cb1fd712ce1bb2e0d037cd06707afbe13e294151a7c8cffe4eb"}
Sep 09 15:24:45.691676 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:45.691624    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-ntzdf" podStartSLOduration=33.856585303 podStartE2EDuration="35.691607532s" totalImagesPullingTime="1.835022229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:24:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:24:45.690151493 +0000 UTC m=+275.332193482" watchObservedRunningTime="2026-09-09 15:24:45.691607532 +0000 UTC m=+275.333649522"
Sep 09 15:24:45.793539 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:45.793495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e6e7e9e2-4a62-47f9-a603-1c77e465e43a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-4zzgj\" (UID: \"e6e7e9e2-4a62-47f9-a603-1c77e465e43a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-4zzgj"
Sep 09 15:24:45.796217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:45.796193    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e6e7e9e2-4a62-47f9-a603-1c77e465e43a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-4zzgj\" (UID: \"e6e7e9e2-4a62-47f9-a603-1c77e465e43a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-4zzgj"
Sep 09 15:24:46.040967 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:46.040866    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-4zzgj"
Sep 09 15:24:46.236149 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:46.236117    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:46.304448 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:46.304367    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-4zzgj"]
Sep 09 15:24:46.308161 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:24:46.308124    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e7e9e2_4a62_47f9_a603_1c77e465e43a.slice/crio-96029edf43d33ef141ba1ebbfbd0252bc8e80e7b475fb5238933afecfebebd3a WatchSource:0}: Error finding container 96029edf43d33ef141ba1ebbfbd0252bc8e80e7b475fb5238933afecfebebd3a: Status 404 returned error can't find the container with id 96029edf43d33ef141ba1ebbfbd0252bc8e80e7b475fb5238933afecfebebd3a
Sep 09 15:24:46.677687 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:46.677658    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-4zzgj" event={"ID":"e6e7e9e2-4a62-47f9-a603-1c77e465e43a","Type":"ContainerStarted","Data":"96029edf43d33ef141ba1ebbfbd0252bc8e80e7b475fb5238933afecfebebd3a"}
Sep 09 15:24:47.416298 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:47.416264    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:47.506385 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:47.506347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:47.508627 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:47.508607    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da71af-045e-45d6-adc6-dca483128c4f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xnkml\" (UID: \"85da71af-045e-45d6-adc6-dca483128c4f\") " pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:47.682195 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:47.682123    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-4zzgj" event={"ID":"e6e7e9e2-4a62-47f9-a603-1c77e465e43a","Type":"ContainerStarted","Data":"12d8884cc6138552473b38fdd677fe263c3d1b952274c5ea67786ab340bdf58e"}
Sep 09 15:24:47.682328 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:47.682249    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-4zzgj"
Sep 09 15:24:47.686710 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:47.686690    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-4zzgj"
Sep 09 15:24:47.696440 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:47.696297    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-4zzgj" podStartSLOduration=1.4536566579999999 podStartE2EDuration="2.69627909s" totalImagesPullingTime="1.242622432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:24:47.696220593 +0000 UTC m=+277.338262582" watchObservedRunningTime="2026-09-09 15:24:47.69627909 +0000 UTC m=+277.338321074"
Sep 09 15:24:47.742080 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:47.742052    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xnkml"
Sep 09 15:24:48.011044 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.010955    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xnkml"]
Sep 09 15:24:48.014609 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:24:48.014571    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85da71af_045e_45d6_adc6_dca483128c4f.slice/crio-e9fa6b18ed66d3c7ce3bc5f43a4260c683e20c1dfd708fbd12ce3009507c3dc2 WatchSource:0}: Error finding container e9fa6b18ed66d3c7ce3bc5f43a4260c683e20c1dfd708fbd12ce3009507c3dc2: Status 404 returned error can't find the container with id e9fa6b18ed66d3c7ce3bc5f43a4260c683e20c1dfd708fbd12ce3009507c3dc2
Sep 09 15:24:48.149545 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.149511    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:48.180188 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.180161    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-txjcq"]
Sep 09 15:24:48.183336 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.183320    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-txjcq"
Sep 09 15:24:48.186558 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.186536    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-87t8r\""
Sep 09 15:24:48.186658 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.186571    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 15:24:48.186658 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.186613    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 15:24:48.186658 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.186619    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 15:24:48.192549 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.192530    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-txjcq"]
Sep 09 15:24:48.210603 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.210579    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:48.211695 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.211678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g8wz\" (UniqueName: \"kubernetes.io/projected/ed66c965-aa5c-41e1-b2ba-6e9abe252472-kube-api-access-2g8wz\") pod \"prometheus-operator-6f4946db56-txjcq\" (UID: \"ed66c965-aa5c-41e1-b2ba-6e9abe252472\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-txjcq"
Sep 09 15:24:48.211749 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.211704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed66c965-aa5c-41e1-b2ba-6e9abe252472-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-txjcq\" (UID: \"ed66c965-aa5c-41e1-b2ba-6e9abe252472\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-txjcq"
Sep 09 15:24:48.211749 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.211740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed66c965-aa5c-41e1-b2ba-6e9abe252472-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-txjcq\" (UID: \"ed66c965-aa5c-41e1-b2ba-6e9abe252472\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-txjcq"
Sep 09 15:24:48.211844 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.211828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed66c965-aa5c-41e1-b2ba-6e9abe252472-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-txjcq\" (UID: \"ed66c965-aa5c-41e1-b2ba-6e9abe252472\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-txjcq"
Sep 09 15:24:48.286391 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.286314    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:48.312533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.312510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g8wz\" (UniqueName: \"kubernetes.io/projected/ed66c965-aa5c-41e1-b2ba-6e9abe252472-kube-api-access-2g8wz\") pod \"prometheus-operator-6f4946db56-txjcq\" (UID: \"ed66c965-aa5c-41e1-b2ba-6e9abe252472\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-txjcq"
Sep 09 15:24:48.312633 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.312540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed66c965-aa5c-41e1-b2ba-6e9abe252472-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-txjcq\" (UID: \"ed66c965-aa5c-41e1-b2ba-6e9abe252472\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-txjcq"
Sep 09 15:24:48.312633 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.312575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed66c965-aa5c-41e1-b2ba-6e9abe252472-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-txjcq\" (UID: \"ed66c965-aa5c-41e1-b2ba-6e9abe252472\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-txjcq"
Sep 09 15:24:48.312633 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.312610    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed66c965-aa5c-41e1-b2ba-6e9abe252472-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-txjcq\" (UID: \"ed66c965-aa5c-41e1-b2ba-6e9abe252472\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-txjcq"
Sep 09 15:24:48.313167 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.313136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed66c965-aa5c-41e1-b2ba-6e9abe252472-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-txjcq\" (UID: \"ed66c965-aa5c-41e1-b2ba-6e9abe252472\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-txjcq"
Sep 09 15:24:48.315154 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.315124    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed66c965-aa5c-41e1-b2ba-6e9abe252472-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-txjcq\" (UID: \"ed66c965-aa5c-41e1-b2ba-6e9abe252472\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-txjcq"
Sep 09 15:24:48.315235 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.315185    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed66c965-aa5c-41e1-b2ba-6e9abe252472-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-txjcq\" (UID: \"ed66c965-aa5c-41e1-b2ba-6e9abe252472\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-txjcq"
Sep 09 15:24:48.319835 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.319818    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g8wz\" (UniqueName: \"kubernetes.io/projected/ed66c965-aa5c-41e1-b2ba-6e9abe252472-kube-api-access-2g8wz\") pod \"prometheus-operator-6f4946db56-txjcq\" (UID: \"ed66c965-aa5c-41e1-b2ba-6e9abe252472\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-txjcq"
Sep 09 15:24:48.492894 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.492847    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-txjcq"
Sep 09 15:24:48.686967 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.686933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xnkml" event={"ID":"85da71af-045e-45d6-adc6-dca483128c4f","Type":"ContainerStarted","Data":"d599530d97894d901cb98b3b74fbc61bb8708b03ad87a57dab0f70e4cc30310f"}
Sep 09 15:24:48.687145 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.687006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xnkml" event={"ID":"85da71af-045e-45d6-adc6-dca483128c4f","Type":"ContainerStarted","Data":"e9fa6b18ed66d3c7ce3bc5f43a4260c683e20c1dfd708fbd12ce3009507c3dc2"}
Sep 09 15:24:48.758151 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.757854    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-txjcq"]
Sep 09 15:24:48.760512 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:24:48.760478    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded66c965_aa5c_41e1_b2ba_6e9abe252472.slice/crio-708c890690dccaccbc7e5a5df28005f8287f4366237d9e5d8f726549939bdc60 WatchSource:0}: Error finding container 708c890690dccaccbc7e5a5df28005f8287f4366237d9e5d8f726549939bdc60: Status 404 returned error can't find the container with id 708c890690dccaccbc7e5a5df28005f8287f4366237d9e5d8f726549939bdc60
Sep 09 15:24:48.864585 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.864512    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:48.965643 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:48.965612    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:49.691566 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:49.691525    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-txjcq" event={"ID":"ed66c965-aa5c-41e1-b2ba-6e9abe252472","Type":"ContainerStarted","Data":"708c890690dccaccbc7e5a5df28005f8287f4366237d9e5d8f726549939bdc60"}
Sep 09 15:24:50.141030 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:50.140994    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:50.352688 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:50.352600    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq" podUID="4637ce97-9134-469e-9b3c-388508a3858b"
Sep 09 15:24:50.352688 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:50.352601    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-62m26" podUID="18973d3c-7fa4-4b71-ade6-1cd976413494"
Sep 09 15:24:50.352688 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:50.352601    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-rdv4c" podUID="ed80dd24-f513-4215-9f41-c33847a78c80"
Sep 09 15:24:50.695903 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:50.695870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-txjcq" event={"ID":"ed66c965-aa5c-41e1-b2ba-6e9abe252472","Type":"ContainerStarted","Data":"7a093995238a88affbd1400cc7f516675106913b5f73591d7233dac3d9d208f6"}
Sep 09 15:24:50.696348 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:50.695924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-txjcq" event={"ID":"ed66c965-aa5c-41e1-b2ba-6e9abe252472","Type":"ContainerStarted","Data":"c05f3afd5c7c46d6b3b0a355fd0d929fc9b71c8cbcd9487513123f41fcc34794"}
Sep 09 15:24:50.697537 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:50.697520    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:24:50.697593 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:50.697564    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xnkml" event={"ID":"85da71af-045e-45d6-adc6-dca483128c4f","Type":"ContainerStarted","Data":"a8c761ed7162af57081b119a30df33e3d4db1b4e1a1a13f1d4f9dcc1cd09bc0c"}
Sep 09 15:24:50.697693 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:50.697680    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:24:50.697817 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:50.697806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:24:50.711603 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:50.711557    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-txjcq" podStartSLOduration=1.588788678 podStartE2EDuration="2.711543704s" totalImagesPullingTime="1.122755026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:24:50.710102085 +0000 UTC m=+280.352144074" watchObservedRunningTime="2026-09-09 15:24:50.711543704 +0000 UTC m=+280.353585693"
Sep 09 15:24:52.293562 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.293481    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:52.428648 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.428617    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:52.525332 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.525302    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:52.556763 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.556694    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-66kk7"]
Sep 09 15:24:52.559870 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.559854    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.562398 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.562377    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7vtq5\""
Sep 09 15:24:52.562529 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.562402    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 15:24:52.562591 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.562540    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 15:24:52.562727 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.562712    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 15:24:52.641960 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.641929    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:52.648076 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.648057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e455fe37-13c1-47ff-bc2c-71730947cea9-node-exporter-textfile\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.648177 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.648104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e455fe37-13c1-47ff-bc2c-71730947cea9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.648177 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.648125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e455fe37-13c1-47ff-bc2c-71730947cea9-node-exporter-tls\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.648177 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.648140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2d84\" (UniqueName: \"kubernetes.io/projected/e455fe37-13c1-47ff-bc2c-71730947cea9-kube-api-access-t2d84\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.648284 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.648185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e455fe37-13c1-47ff-bc2c-71730947cea9-sys\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.648284 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.648200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e455fe37-13c1-47ff-bc2c-71730947cea9-metrics-client-ca\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.648284 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.648221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e455fe37-13c1-47ff-bc2c-71730947cea9-node-exporter-wtmp\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.648284 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.648248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e455fe37-13c1-47ff-bc2c-71730947cea9-root\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.648284 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.648274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e455fe37-13c1-47ff-bc2c-71730947cea9-node-exporter-accelerators-collector-config\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.706165 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.706133    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xnkml" event={"ID":"85da71af-045e-45d6-adc6-dca483128c4f","Type":"ContainerStarted","Data":"c8d17720ca3b0c15de363cc16d69e26c518de03a130469f4d0b3e52f550f024f"}
Sep 09 15:24:52.749542 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.749514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e455fe37-13c1-47ff-bc2c-71730947cea9-node-exporter-tls\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.749542 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.749541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2d84\" (UniqueName: \"kubernetes.io/projected/e455fe37-13c1-47ff-bc2c-71730947cea9-kube-api-access-t2d84\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.749751 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.749563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e455fe37-13c1-47ff-bc2c-71730947cea9-sys\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.749751 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.749578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e455fe37-13c1-47ff-bc2c-71730947cea9-metrics-client-ca\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.749751 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.749602    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e455fe37-13c1-47ff-bc2c-71730947cea9-node-exporter-wtmp\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.749751 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.749628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e455fe37-13c1-47ff-bc2c-71730947cea9-root\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.749751 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.749648    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e455fe37-13c1-47ff-bc2c-71730947cea9-sys\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.749751 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:52.749665    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 15:24:52.749751 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.749672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e455fe37-13c1-47ff-bc2c-71730947cea9-node-exporter-accelerators-collector-config\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.749751 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.749714    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e455fe37-13c1-47ff-bc2c-71730947cea9-root\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.749751 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:52.749726    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e455fe37-13c1-47ff-bc2c-71730947cea9-node-exporter-tls podName:e455fe37-13c1-47ff-bc2c-71730947cea9 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:53.249708454 +0000 UTC m=+282.891750426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e455fe37-13c1-47ff-bc2c-71730947cea9-node-exporter-tls") pod "node-exporter-66kk7" (UID: "e455fe37-13c1-47ff-bc2c-71730947cea9") : secret "node-exporter-tls" not found
Sep 09 15:24:52.750255 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.749762    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e455fe37-13c1-47ff-bc2c-71730947cea9-node-exporter-wtmp\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.750255 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.749784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e455fe37-13c1-47ff-bc2c-71730947cea9-node-exporter-textfile\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.750255 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.749862    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e455fe37-13c1-47ff-bc2c-71730947cea9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.750255 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.750048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e455fe37-13c1-47ff-bc2c-71730947cea9-node-exporter-textfile\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.750255 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.750241    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e455fe37-13c1-47ff-bc2c-71730947cea9-metrics-client-ca\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.750515 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.750371    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e455fe37-13c1-47ff-bc2c-71730947cea9-node-exporter-accelerators-collector-config\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.752197 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.752180    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e455fe37-13c1-47ff-bc2c-71730947cea9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.760146 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.760126    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2d84\" (UniqueName: \"kubernetes.io/projected/e455fe37-13c1-47ff-bc2c-71730947cea9-kube-api-access-t2d84\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:52.810571 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.810501    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:52.876626 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.876599    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:52.978458 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:52.978423    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:53.078918 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.078829    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:53.179392 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.179362    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:53.254268 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.254235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e455fe37-13c1-47ff-bc2c-71730947cea9-node-exporter-tls\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:53.256726 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.256703    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e455fe37-13c1-47ff-bc2c-71730947cea9-node-exporter-tls\") pod \"node-exporter-66kk7\" (UID: \"e455fe37-13c1-47ff-bc2c-71730947cea9\") " pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:53.469247 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.469202    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-66kk7"
Sep 09 15:24:53.481483 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:24:53.481451    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode455fe37_13c1_47ff_bc2c_71730947cea9.slice/crio-452fb7de037cd7ea524d4a3a5eec6993e9a9ae339d205fe7f530106d45ceba67 WatchSource:0}: Error finding container 452fb7de037cd7ea524d4a3a5eec6993e9a9ae339d205fe7f530106d45ceba67: Status 404 returned error can't find the container with id 452fb7de037cd7ea524d4a3a5eec6993e9a9ae339d205fe7f530106d45ceba67
Sep 09 15:24:53.529713 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.529681    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:53.608559 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.608529    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:53.613628 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.613574    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xnkml" podStartSLOduration=34.719268357 podStartE2EDuration="38.613554301s" totalImagesPullingTime="3.894285944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:24:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:24:52.722260324 +0000 UTC m=+282.364302317" watchObservedRunningTime="2026-09-09 15:24:53.613554301 +0000 UTC m=+283.255596291"
Sep 09 15:24:53.614927 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.614905    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 15:24:53.620348 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.620328    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.622737 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.622716    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 15:24:53.622834 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.622738    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8gt4p\""
Sep 09 15:24:53.623246 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.623231    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 15:24:53.623322 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.623307    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 15:24:53.623455 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.623435    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 15:24:53.623539 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.623524    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 15:24:53.623597 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.623543    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 15:24:53.623597 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.623587    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 15:24:53.623696 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.623661    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 15:24:53.623861 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.623843    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 15:24:53.631326 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.631307    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 15:24:53.658085 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.658053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-config-volume\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.658212 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.658119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6686e44e-ff16-458c-ad0c-3b65eec595e1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.658287 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.658214    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rct5p\" (UniqueName: \"kubernetes.io/projected/6686e44e-ff16-458c-ad0c-3b65eec595e1-kube-api-access-rct5p\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.658287 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.658248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6686e44e-ff16-458c-ad0c-3b65eec595e1-config-out\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.658287 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.658283    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.658448 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.658333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-web-config\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.658520 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.658498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.658590 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.658533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.658590 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.658567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6686e44e-ff16-458c-ad0c-3b65eec595e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.658694 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.658602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6686e44e-ff16-458c-ad0c-3b65eec595e1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.658694 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.658641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6686e44e-ff16-458c-ad0c-3b65eec595e1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.658783 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.658715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.658783 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.658775    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.710817 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.710789    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66kk7" event={"ID":"e455fe37-13c1-47ff-bc2c-71730947cea9","Type":"ContainerStarted","Data":"452fb7de037cd7ea524d4a3a5eec6993e9a9ae339d205fe7f530106d45ceba67"}
Sep 09 15:24:53.724603 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.724523    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:53.759806 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.759778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rct5p\" (UniqueName: \"kubernetes.io/projected/6686e44e-ff16-458c-ad0c-3b65eec595e1-kube-api-access-rct5p\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.759942 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.759805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6686e44e-ff16-458c-ad0c-3b65eec595e1-config-out\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.759942 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.759839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.759942 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.759872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:24:53.759942 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.759887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-web-config\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.760324 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.759946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.760324 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.759971    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.760324 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.760018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6686e44e-ff16-458c-ad0c-3b65eec595e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.760324 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.760051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6686e44e-ff16-458c-ad0c-3b65eec595e1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.760324 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.760078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-62m26\" (UID: \"18973d3c-7fa4-4b71-ade6-1cd976413494\") " pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:24:53.760324 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.760103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6686e44e-ff16-458c-ad0c-3b65eec595e1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.760324 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.760154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.760324 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.760181    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:24:53.760324 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.760233    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.760324 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.760261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-config-volume\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.760324 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.760289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6686e44e-ff16-458c-ad0c-3b65eec595e1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.761035 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.760626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6686e44e-ff16-458c-ad0c-3b65eec595e1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.761737 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.761397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6686e44e-ff16-458c-ad0c-3b65eec595e1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.762368 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:53.762278    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6686e44e-ff16-458c-ad0c-3b65eec595e1-alertmanager-trusted-ca-bundle podName:6686e44e-ff16-458c-ad0c-3b65eec595e1 nodeName:}" failed. No retries permitted until 2026-09-09 15:24:54.26225716 +0000 UTC m=+283.904299148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/6686e44e-ff16-458c-ad0c-3b65eec595e1-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "6686e44e-ff16-458c-ad0c-3b65eec595e1") : configmap references non-existent config key: ca-bundle.crt
Sep 09 15:24:53.763492 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.763463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6686e44e-ff16-458c-ad0c-3b65eec595e1-config-out\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.763575 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.763470    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.763575 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.763533    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.764130 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.764106    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed80dd24-f513-4215-9f41-c33847a78c80-metrics-tls\") pod \"dns-default-rdv4c\" (UID: \"ed80dd24-f513-4215-9f41-c33847a78c80\") " pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:24:53.765693 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.765657    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-web-config\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.765787 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.765760    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.765847 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.765781    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.765953 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.765926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18973d3c-7fa4-4b71-ade6-1cd976413494-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-62m26\" (UID: \"18973d3c-7fa4-4b71-ade6-1cd976413494\") " pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:24:53.766215 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.766168    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.766477 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.766449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6686e44e-ff16-458c-ad0c-3b65eec595e1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.766710 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.766679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-config-volume\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:53.767791 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.767774    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls\") pod \"image-registry-8945c7fd4-h4fdq\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") " pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:24:53.768047 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:53.768029    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rct5p\" (UniqueName: \"kubernetes.io/projected/6686e44e-ff16-458c-ad0c-3b65eec595e1-kube-api-access-rct5p\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:54.001206 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.001115    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jnqpd\""
Sep 09 15:24:54.001206 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.001182    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hz8wm\""
Sep 09 15:24:54.001431 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.001182    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pzhsz\""
Sep 09 15:24:54.009139 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.009112    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:24:54.009248 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.009144    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-62m26"
Sep 09 15:24:54.009248 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.009126    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:24:54.265475 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.265404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6686e44e-ff16-458c-ad0c-3b65eec595e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:54.266245 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.266228    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6686e44e-ff16-458c-ad0c-3b65eec595e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:54.294913 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.294889    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rdv4c"]
Sep 09 15:24:54.297146 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:24:54.297112    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded80dd24_f513_4215_9f41_c33847a78c80.slice/crio-b78267c409289cdec07d4fff3a5de9084589d0ff5486f22026a8a64ce2b08d7e WatchSource:0}: Error finding container b78267c409289cdec07d4fff3a5de9084589d0ff5486f22026a8a64ce2b08d7e: Status 404 returned error can't find the container with id b78267c409289cdec07d4fff3a5de9084589d0ff5486f22026a8a64ce2b08d7e
Sep 09 15:24:54.301970 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.301948    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-62m26"]
Sep 09 15:24:54.304727 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.304704    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8945c7fd4-h4fdq"]
Sep 09 15:24:54.305267 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:24:54.305232    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18973d3c_7fa4_4b71_ade6_1cd976413494.slice/crio-d303e175c8c5cb357eadd485b2b05cfd38968ea0e8e4a5de695fb9256c1993ac WatchSource:0}: Error finding container d303e175c8c5cb357eadd485b2b05cfd38968ea0e8e4a5de695fb9256c1993ac: Status 404 returned error can't find the container with id d303e175c8c5cb357eadd485b2b05cfd38968ea0e8e4a5de695fb9256c1993ac
Sep 09 15:24:54.307755 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:24:54.307733    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4637ce97_9134_469e_9b3c_388508a3858b.slice/crio-d73c67c699885573f409e594d2aab7794182c7c1f9a98d89466fc955b529a6c5 WatchSource:0}: Error finding container d73c67c699885573f409e594d2aab7794182c7c1f9a98d89466fc955b529a6c5: Status 404 returned error can't find the container with id d73c67c699885573f409e594d2aab7794182c7c1f9a98d89466fc955b529a6c5
Sep 09 15:24:54.531557 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.531485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:24:54.716486 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.716439    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq" event={"ID":"4637ce97-9134-469e-9b3c-388508a3858b","Type":"ContainerStarted","Data":"52826c00b9a2eca2d330eb74b0206d62da23d822d8bac7a583971d16120d713a"}
Sep 09 15:24:54.716659 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.716520    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq" event={"ID":"4637ce97-9134-469e-9b3c-388508a3858b","Type":"ContainerStarted","Data":"d73c67c699885573f409e594d2aab7794182c7c1f9a98d89466fc955b529a6c5"}
Sep 09 15:24:54.716659 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.716574    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:24:54.717928 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.717905    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-62m26" event={"ID":"18973d3c-7fa4-4b71-ade6-1cd976413494","Type":"ContainerStarted","Data":"d303e175c8c5cb357eadd485b2b05cfd38968ea0e8e4a5de695fb9256c1993ac"}
Sep 09 15:24:54.719252 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.719228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rdv4c" event={"ID":"ed80dd24-f513-4215-9f41-c33847a78c80","Type":"ContainerStarted","Data":"b78267c409289cdec07d4fff3a5de9084589d0ff5486f22026a8a64ce2b08d7e"}
Sep 09 15:24:54.747952 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.747881    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq" podStartSLOduration=283.747863034 podStartE2EDuration="4m43.747863034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:24:54.746626836 +0000 UTC m=+284.388668824" watchObservedRunningTime="2026-09-09 15:24:54.747863034 +0000 UTC m=+284.389905023"
Sep 09 15:24:54.814874 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.814796    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 15:24:54.818541 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:24:54.818508    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6686e44e_ff16_458c_ad0c_3b65eec595e1.slice/crio-9a8f3fd1c5aa51f919d0e26c978f6b11f196eb920eb62b645da87896c58b030a WatchSource:0}: Error finding container 9a8f3fd1c5aa51f919d0e26c978f6b11f196eb920eb62b645da87896c58b030a: Status 404 returned error can't find the container with id 9a8f3fd1c5aa51f919d0e26c978f6b11f196eb920eb62b645da87896c58b030a
Sep 09 15:24:54.861742 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:54.861711    2578 scope.go:117] "RemoveContainer" containerID="c7474639070de4ae23a6a00efdfc7687aedc2a5a8630e83020b40ba42301c3c6"
Sep 09 15:24:55.486691 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:55.486609    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:55.637888 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:55.637815    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:55.724794 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:55.724763    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6686e44e-ff16-458c-ad0c-3b65eec595e1","Type":"ContainerStarted","Data":"9a8f3fd1c5aa51f919d0e26c978f6b11f196eb920eb62b645da87896c58b030a"}
Sep 09 15:24:55.726580 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:55.726561    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 15:24:55.726711 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:55.726645    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-vsgld" event={"ID":"dc2e93bc-cc28-454c-9526-13e2837cdd14","Type":"ContainerStarted","Data":"db8391fb8fa6861019b7592bb88f0ac19b714b76543e425c21d7a9c04b56a738"}
Sep 09 15:24:55.727054 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:55.727015    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:55.742734 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:55.742662    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-vsgld" podStartSLOduration=43.251590569 podStartE2EDuration="45.742643193s" totalImagesPullingTime="2.491052624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:24:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:24:55.741900903 +0000 UTC m=+285.383942891" watchObservedRunningTime="2026-09-09 15:24:55.742643193 +0000 UTC m=+285.384685183"
Sep 09 15:24:55.848328 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:55.848294    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:56.348734 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.348701    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:56.357453 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.357352    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-vsgld"
Sep 09 15:24:56.531192 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.531159    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-ll79v"]
Sep 09 15:24:56.535641 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.535619    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-ll79v"
Sep 09 15:24:56.538151 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.538130    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7cnc9\""
Sep 09 15:24:56.538478 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.538456    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 15:24:56.538582 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.538528    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 15:24:56.545624 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.545563    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-ll79v"]
Sep 09 15:24:56.563395 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.563370    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:56.590136 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.590102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtq8w\" (UniqueName: \"kubernetes.io/projected/22748ed4-a6ec-4cad-a2a6-f3a10c1d52a4-kube-api-access-vtq8w\") pod \"downloads-85d98c998f-ll79v\" (UID: \"22748ed4-a6ec-4cad-a2a6-f3a10c1d52a4\") " pod="openshift-console/downloads-85d98c998f-ll79v"
Sep 09 15:24:56.642316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.642288    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:56.691108 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.691088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtq8w\" (UniqueName: \"kubernetes.io/projected/22748ed4-a6ec-4cad-a2a6-f3a10c1d52a4-kube-api-access-vtq8w\") pod \"downloads-85d98c998f-ll79v\" (UID: \"22748ed4-a6ec-4cad-a2a6-f3a10c1d52a4\") " pod="openshift-console/downloads-85d98c998f-ll79v"
Sep 09 15:24:56.701132 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.701106    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtq8w\" (UniqueName: \"kubernetes.io/projected/22748ed4-a6ec-4cad-a2a6-f3a10c1d52a4-kube-api-access-vtq8w\") pod \"downloads-85d98c998f-ll79v\" (UID: \"22748ed4-a6ec-4cad-a2a6-f3a10c1d52a4\") " pod="openshift-console/downloads-85d98c998f-ll79v"
Sep 09 15:24:56.736740 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.736704    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66kk7" event={"ID":"e455fe37-13c1-47ff-bc2c-71730947cea9","Type":"ContainerStarted","Data":"2a522ed326b0546894ddaf8c09684f7fdeca4469907149dc758a0813694fb5f1"}
Sep 09 15:24:56.739273 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.739124    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-62m26" event={"ID":"18973d3c-7fa4-4b71-ade6-1cd976413494","Type":"ContainerStarted","Data":"d643f984d544ced2eadb48e4b47587a540924d60c0108d324746c3fa60b61abc"}
Sep 09 15:24:56.741566 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.741539    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rdv4c" event={"ID":"ed80dd24-f513-4215-9f41-c33847a78c80","Type":"ContainerStarted","Data":"a9a00bf16c0b6f4989c98c6141373f99b3962d4bbb4ac26cb5a1bcb8ba9ea5e5"}
Sep 09 15:24:56.745467 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.745444    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6686e44e-ff16-458c-ad0c-3b65eec595e1","Type":"ContainerStarted","Data":"e391698c3c4fd10898c1259912bedd5cc03a22b3626837c4738a029601ec4ff7"}
Sep 09 15:24:56.805445 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.805394    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-62m26" podStartSLOduration=279.513297769 podStartE2EDuration="4m41.805378341s" totalImagesPullingTime="2.292080572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:24:56.805180953 +0000 UTC m=+286.447222942" watchObservedRunningTime="2026-09-09 15:24:56.805378341 +0000 UTC m=+286.447420386"
Sep 09 15:24:56.844017 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.843968    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:56.847922 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.847870    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-ll79v"
Sep 09 15:24:56.943334 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:56.943292    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:57.120631 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.120543    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-ll79v"]
Sep 09 15:24:57.123506 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:24:57.123474    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22748ed4_a6ec_4cad_a2a6_f3a10c1d52a4.slice/crio-4bcecaafcbb2e126c9882e1826c6fab3510124d98ad6467dfe008772dc4b2611 WatchSource:0}: Error finding container 4bcecaafcbb2e126c9882e1826c6fab3510124d98ad6467dfe008772dc4b2611: Status 404 returned error can't find the container with id 4bcecaafcbb2e126c9882e1826c6fab3510124d98ad6467dfe008772dc4b2611
Sep 09 15:24:57.146325 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.146299    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:57.211094 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.211070    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:57.300738 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.300713    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:57.318260 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.318230    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-2l7x5"]
Sep 09 15:24:57.322827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.322809    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2l7x5"
Sep 09 15:24:57.324879 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.324856    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 09 15:24:57.324971 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.324935    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-fzwmw\""
Sep 09 15:24:57.328842 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.328819    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-2l7x5"]
Sep 09 15:24:57.359720 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.359692    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:57.397851 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.397819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-2l7x5\" (UID: \"2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2l7x5"
Sep 09 15:24:57.398029 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.397872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sft7\" (UniqueName: \"kubernetes.io/projected/2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc-kube-api-access-4sft7\") pod \"monitoring-plugin-85bbbbbf4-2l7x5\" (UID: \"2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2l7x5"
Sep 09 15:24:57.499286 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.499255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sft7\" (UniqueName: \"kubernetes.io/projected/2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc-kube-api-access-4sft7\") pod \"monitoring-plugin-85bbbbbf4-2l7x5\" (UID: \"2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2l7x5"
Sep 09 15:24:57.499425 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.499333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-2l7x5\" (UID: \"2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2l7x5"
Sep 09 15:24:57.499425 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:57.499420    2578 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 09 15:24:57.499501 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:57.499471    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc-monitoring-plugin-cert podName:2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc nodeName:}" failed. No retries permitted until 2026-09-09 15:24:57.999456324 +0000 UTC m=+287.641498292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-2l7x5" (UID: "2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc") : secret "monitoring-plugin-cert" not found
Sep 09 15:24:57.507578 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.507556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sft7\" (UniqueName: \"kubernetes.io/projected/2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc-kube-api-access-4sft7\") pod \"monitoring-plugin-85bbbbbf4-2l7x5\" (UID: \"2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2l7x5"
Sep 09 15:24:57.615187 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.615156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:57.750794 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.750714    2578 generic.go:356] "Generic (PLEG): container finished" podID="e455fe37-13c1-47ff-bc2c-71730947cea9" containerID="2a522ed326b0546894ddaf8c09684f7fdeca4469907149dc758a0813694fb5f1" exitCode=0
Sep 09 15:24:57.751424 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.750806    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66kk7" event={"ID":"e455fe37-13c1-47ff-bc2c-71730947cea9","Type":"ContainerDied","Data":"2a522ed326b0546894ddaf8c09684f7fdeca4469907149dc758a0813694fb5f1"}
Sep 09 15:24:57.753313 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.753288    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rdv4c" event={"ID":"ed80dd24-f513-4215-9f41-c33847a78c80","Type":"ContainerStarted","Data":"bbf57176d54c202b18f6c8a15edae7a5a19ae4665e1460dc009d03d068f94d2e"}
Sep 09 15:24:57.753431 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.753365    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:24:57.755189 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.755163    2578 generic.go:356] "Generic (PLEG): container finished" podID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerID="e391698c3c4fd10898c1259912bedd5cc03a22b3626837c4738a029601ec4ff7" exitCode=0
Sep 09 15:24:57.755323 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.755249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6686e44e-ff16-458c-ad0c-3b65eec595e1","Type":"ContainerDied","Data":"e391698c3c4fd10898c1259912bedd5cc03a22b3626837c4738a029601ec4ff7"}
Sep 09 15:24:57.756664 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.756636    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-ll79v" event={"ID":"22748ed4-a6ec-4cad-a2a6-f3a10c1d52a4","Type":"ContainerStarted","Data":"4bcecaafcbb2e126c9882e1826c6fab3510124d98ad6467dfe008772dc4b2611"}
Sep 09 15:24:57.786460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.786210    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rdv4c" podStartSLOduration=252.48607625 podStartE2EDuration="4m14.786191939s" totalImagesPullingTime="2.300115689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:24:57.786057957 +0000 UTC m=+287.428099958" watchObservedRunningTime="2026-09-09 15:24:57.786191939 +0000 UTC m=+287.428233929"
Sep 09 15:24:57.883275 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:57.883247    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:58.003580 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.003493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-2l7x5\" (UID: \"2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2l7x5"
Sep 09 15:24:58.003757 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:58.003652    2578 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 09 15:24:58.003757 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:24:58.003708    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc-monitoring-plugin-cert podName:2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc nodeName:}" failed. No retries permitted until 2026-09-09 15:24:59.003692468 +0000 UTC m=+288.645734448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-2l7x5" (UID: "2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc") : secret "monitoring-plugin-cert" not found
Sep 09 15:24:58.027353 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.027301    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:58.131618 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.131567    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:58.229262 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.229224    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:58.287742 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.287669    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:58.425686 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.425660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:58.480410 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.480372    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:58.765679 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.765640    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66kk7" event={"ID":"e455fe37-13c1-47ff-bc2c-71730947cea9","Type":"ContainerStarted","Data":"3560d8a91585435045d052c85c79548ddedcf70f8690c000b11f922adcfcdbb8"}
Sep 09 15:24:58.766201 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.765737    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66kk7" event={"ID":"e455fe37-13c1-47ff-bc2c-71730947cea9","Type":"ContainerStarted","Data":"bd2af76cb669a7441a61741d67adb6e26ebe1c0bfd39c79e8f628a9f7418b8c2"}
Sep 09 15:24:58.787171 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.787125    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:24:58.792482 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.790269    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 15:24:58.792482 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.791614    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-66kk7" podStartSLOduration=3.582897864 podStartE2EDuration="6.791573076s" totalImagesPullingTime="3.115976894s" totalInitContainerRuntime="92.698318ms" isStatefulPod=true podCreationTimestamp="2026-09-09 15:24:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:24:58.787917014 +0000 UTC m=+288.429959006" watchObservedRunningTime="2026-09-09 15:24:58.791573076 +0000 UTC m=+288.433615067"
Sep 09 15:24:58.797770 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.797725    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.802622 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.801257    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 15:24:58.802622 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.801432    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 15:24:58.802622 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.801454    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 15:24:58.802622 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.801685    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 15:24:58.802622 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.801784    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 15:24:58.802622 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.801857    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 15:24:58.802622 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.801959    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 15:24:58.802622 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.802034    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fh66r83d9f5bt\""
Sep 09 15:24:58.802622 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.802196    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 15:24:58.802622 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.802359    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 15:24:58.802622 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.802437    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 15:24:58.802622 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.801689    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 15:24:58.803296 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.802652    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 15:24:58.803296 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.802667    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 15:24:58.803296 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.802819    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 15:24:58.803296 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.803011    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6xfj9\""
Sep 09 15:24:58.804776 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.804759    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 15:24:58.808743 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.808720    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 15:24:58.912425 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.912276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.912425 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.912385    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.912673 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.912436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.912673 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.912483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf262\" (UniqueName: \"kubernetes.io/projected/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-kube-api-access-wf262\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.912673 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.912515    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.912673 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.912538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.912673 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.912563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.912892 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.912670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.912892 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.912709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-config\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.912892 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.912745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.912892 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.912831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.912892 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.912884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.913140 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.912937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-config-out\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.913140 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.912959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-web-config\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.913140 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.913006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.913140 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.913037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.913140 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.913081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.913140 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.913104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.913402 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.913164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:58.913402 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:58.913207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.014599 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.014561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.014599 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.014611    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.014865 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.014642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-config-out\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.014865 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.014665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-web-config\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.014865 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.014711    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.014865 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.014743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.014865 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.014771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.014865 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.014795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.014865 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.014839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.015234 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.014878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.015234 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.014912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.015234 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.014945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.015234 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.015003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-2l7x5\" (UID: \"2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2l7x5"
Sep 09 15:24:59.015234 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.015037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.015234 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.015074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf262\" (UniqueName: \"kubernetes.io/projected/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-kube-api-access-wf262\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.015234 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.015107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.015234 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.015130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.016039 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.015819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.017391 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.017359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.017501 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.017417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.017501 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.017448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-config\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.017501 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.017497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.021461 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.017747    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.021461 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.018016    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.021461 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.018133    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.021461 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.018188    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.021461 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.019921    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.021461 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.020547    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.021461 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.020858    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.021461 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.021315    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.021461 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.021348    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.022831 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.022160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.022831 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.022524    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.022831 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.022761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.023927 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.023904    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-config-out\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.024390 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.024369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.024480 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.024395    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-2l7x5\" (UID: \"2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2l7x5"
Sep 09 15:24:59.024785 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.024743    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-web-config\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.024872 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.024849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.025587 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.025546    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.026748 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.026707    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-config\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.029391 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.029241    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf262\" (UniqueName: \"kubernetes.io/projected/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-kube-api-access-wf262\") pod \"prometheus-k8s-0\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.115335 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.115298    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:24:59.132565 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.132542    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2l7x5"
Sep 09 15:24:59.416037 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.416004    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 15:24:59.418726 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:24:59.418695    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3e59e7_b0e3_4238_93b6_b0531bd7e3aa.slice/crio-96fe5c498555d5052fc6b947701bd69f49189158e54ea11068996cc2ef191ed8 WatchSource:0}: Error finding container 96fe5c498555d5052fc6b947701bd69f49189158e54ea11068996cc2ef191ed8: Status 404 returned error can't find the container with id 96fe5c498555d5052fc6b947701bd69f49189158e54ea11068996cc2ef191ed8
Sep 09 15:24:59.435029 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.435006    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-2l7x5"]
Sep 09 15:24:59.438526 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:24:59.438504    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2070b616_f62d_4ba0_a0ab_eb4bd6a81cbc.slice/crio-d84463de14e288afaa6ae8c703b88ffdac5e09ea3cf4e67ce0bcac60417d389c WatchSource:0}: Error finding container d84463de14e288afaa6ae8c703b88ffdac5e09ea3cf4e67ce0bcac60417d389c: Status 404 returned error can't find the container with id d84463de14e288afaa6ae8c703b88ffdac5e09ea3cf4e67ce0bcac60417d389c
Sep 09 15:24:59.772019 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.771911    2578 generic.go:356] "Generic (PLEG): container finished" podID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerID="f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16" exitCode=0
Sep 09 15:24:59.772389 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.772104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa","Type":"ContainerDied","Data":"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16"}
Sep 09 15:24:59.772389 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.772189    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa","Type":"ContainerStarted","Data":"96fe5c498555d5052fc6b947701bd69f49189158e54ea11068996cc2ef191ed8"}
Sep 09 15:24:59.773707 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:24:59.773572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2l7x5" event={"ID":"2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc","Type":"ContainerStarted","Data":"d84463de14e288afaa6ae8c703b88ffdac5e09ea3cf4e67ce0bcac60417d389c"}
Sep 09 15:25:01.790529 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:01.790220    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6686e44e-ff16-458c-ad0c-3b65eec595e1","Type":"ContainerStarted","Data":"9e217a1d7fac1a7ea777a35cd446a55b22fa8f672f05458c292f9bb0aec708c5"}
Sep 09 15:25:01.790529 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:01.790305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6686e44e-ff16-458c-ad0c-3b65eec595e1","Type":"ContainerStarted","Data":"c3382a8104ee11b63c7169dbc13fc88bbae4531ed5d1ecb52b71e851d54004a3"}
Sep 09 15:25:01.790529 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:01.790353    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6686e44e-ff16-458c-ad0c-3b65eec595e1","Type":"ContainerStarted","Data":"256821f3e6878e29a06e5422441e60914359a22350ce928876d47ff7f78ae338"}
Sep 09 15:25:01.790529 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:01.790394    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6686e44e-ff16-458c-ad0c-3b65eec595e1","Type":"ContainerStarted","Data":"5665056cff09d3a3f51f2a9ea13dca17fc14ea85adc815b494db807c02daec4e"}
Sep 09 15:25:01.790529 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:01.790433    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6686e44e-ff16-458c-ad0c-3b65eec595e1","Type":"ContainerStarted","Data":"f3e70461f78a4a2974285c84818f6a67da6ce0ddb21940c5e92759aa7ea1651a"}
Sep 09 15:25:01.792258 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:01.792226    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2l7x5" event={"ID":"2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc","Type":"ContainerStarted","Data":"3ba0be40d9f5718877219d01e635f6e42e1dfa741549b656e11ab89ea048e020"}
Sep 09 15:25:01.793100 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:01.793080    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2l7x5"
Sep 09 15:25:01.798745 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:01.798714    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2l7x5"
Sep 09 15:25:01.809483 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:01.809429    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2l7x5" podStartSLOduration=2.96253599 podStartE2EDuration="4.809411989s" totalImagesPullingTime="1.846875999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:25:01.80826254 +0000 UTC m=+291.450304530" watchObservedRunningTime="2026-09-09 15:25:01.809411989 +0000 UTC m=+291.451453993"
Sep 09 15:25:03.808058 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:03.808023    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa","Type":"ContainerStarted","Data":"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e"}
Sep 09 15:25:03.808456 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:03.808108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa","Type":"ContainerStarted","Data":"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5"}
Sep 09 15:25:04.818618 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:04.818576    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6686e44e-ff16-458c-ad0c-3b65eec595e1","Type":"ContainerStarted","Data":"986dcc74e7d382a7d9eca69752667fa3e27c301a30c0fa56ba48b643f3231a78"}
Sep 09 15:25:04.844600 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:04.844530    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.098538281 podStartE2EDuration="11.844506678s" totalImagesPullingTime="7.715893392s" totalInitContainerRuntime="30.075005ms" isStatefulPod=false podCreationTimestamp="2026-09-09 15:24:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:25:04.843202525 +0000 UTC m=+294.485244535" watchObservedRunningTime="2026-09-09 15:25:04.844506678 +0000 UTC m=+294.486548668"
Sep 09 15:25:05.862362 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:25:05.862318    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-nhtcj" podUID="7052f7a0-16d5-4115-82ae-34f07e9db9f9"
Sep 09 15:25:06.832350 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:06.832316    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa","Type":"ContainerStarted","Data":"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d"}
Sep 09 15:25:06.832520 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:06.832388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa","Type":"ContainerStarted","Data":"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce"}
Sep 09 15:25:06.832520 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:06.832421    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa","Type":"ContainerStarted","Data":"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a"}
Sep 09 15:25:06.832520 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:06.832446    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa","Type":"ContainerStarted","Data":"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df"}
Sep 09 15:25:06.860999 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:06.860932    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.846648225 podStartE2EDuration="8.860916567s" totalImagesPullingTime="5.991256256s" totalInitContainerRuntime="23.012086ms" isStatefulPod=false podCreationTimestamp="2026-09-09 15:24:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:25:06.858260098 +0000 UTC m=+296.500302080" watchObservedRunningTime="2026-09-09 15:25:06.860916567 +0000 UTC m=+296.502958556"
Sep 09 15:25:07.768264 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:07.768236    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rdv4c"
Sep 09 15:25:08.196604 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:08.196563    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:08.297611 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:08.297570    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:08.348691 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:08.348660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:09.115500 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:09.115448    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:25:09.785593 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:09.785560    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8945c7fd4-h4fdq"]
Sep 09 15:25:09.791304 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:09.791241    2578 patch_prober.go:28] interesting pod/image-registry-8945c7fd4-h4fdq container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 15:25:09.791475 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:09.791311    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq" podUID="4637ce97-9134-469e-9b3c-388508a3858b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 15:25:09.841138 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:09.841102    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:10.843326 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:10.843137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:11.702711 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:11.702672    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:11.757204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:11.757166    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:13.953081 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:13.953038    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:14.123106 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:14.123060    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:14.223793 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:14.223710    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:15.033354 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:15.033329    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 15:25:15.034315 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:15.034294    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 15:25:15.041673 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:15.041654    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:25:15.042747 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:15.042727    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:25:15.045591 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:15.045575    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 15:25:15.877844 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:15.877757    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-ll79v" event={"ID":"22748ed4-a6ec-4cad-a2a6-f3a10c1d52a4","Type":"ContainerStarted","Data":"a371fc703576a3f930b626dc75c614a03c45577bfd7e70af36756f3657cf668f"}
Sep 09 15:25:15.878035 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:15.877949    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-ll79v"
Sep 09 15:25:15.892970 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:15.892944    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-ll79v"
Sep 09 15:25:15.896071 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:15.896025    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-ll79v" podStartSLOduration=1.633107664 podStartE2EDuration="19.896010538s" totalImagesPullingTime="18.262902874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:24:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:25:15.894892752 +0000 UTC m=+305.536934734" watchObservedRunningTime="2026-09-09 15:25:15.896010538 +0000 UTC m=+305.538052529"
Sep 09 15:25:16.885710 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:16.885673    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:16.986874 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:16.986787    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:17.861200 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:17.861166    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:25:19.791348 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:19.791316    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:25:20.447363 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:20.447327    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:20.528860 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:20.528816    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:20.820948 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:20.820869    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:20.873373 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:20.873342    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:22.055552 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:22.055512    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:22.157059 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:22.157027    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:25.516601 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:25.516569    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:25.581261 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:25.581231    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:26.457837 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:26.457806    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:26.558685 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:26.558655    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:34.809654 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:34.809611    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq" podUID="4637ce97-9134-469e-9b3c-388508a3858b" containerName="registry" containerID="cri-o://52826c00b9a2eca2d330eb74b0206d62da23d822d8bac7a583971d16120d713a" gracePeriod=30
Sep 09 15:25:34.957246 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:34.957213    2578 generic.go:356] "Generic (PLEG): container finished" podID="4637ce97-9134-469e-9b3c-388508a3858b" containerID="52826c00b9a2eca2d330eb74b0206d62da23d822d8bac7a583971d16120d713a" exitCode=0
Sep 09 15:25:34.957422 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:34.957249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq" event={"ID":"4637ce97-9134-469e-9b3c-388508a3858b","Type":"ContainerDied","Data":"52826c00b9a2eca2d330eb74b0206d62da23d822d8bac7a583971d16120d713a"}
Sep 09 15:25:35.088372 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.088317    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:25:35.164574 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.164545    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls\") pod \"4637ce97-9134-469e-9b3c-388508a3858b\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") "
Sep 09 15:25:35.164752 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.164607    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4637ce97-9134-469e-9b3c-388508a3858b-ca-trust-extracted\") pod \"4637ce97-9134-469e-9b3c-388508a3858b\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") "
Sep 09 15:25:35.164752 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.164645    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4637ce97-9134-469e-9b3c-388508a3858b-installation-pull-secrets\") pod \"4637ce97-9134-469e-9b3c-388508a3858b\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") "
Sep 09 15:25:35.164752 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.164682    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4637ce97-9134-469e-9b3c-388508a3858b-image-registry-private-configuration\") pod \"4637ce97-9134-469e-9b3c-388508a3858b\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") "
Sep 09 15:25:35.164752 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.164713    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4637ce97-9134-469e-9b3c-388508a3858b-registry-certificates\") pod \"4637ce97-9134-469e-9b3c-388508a3858b\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") "
Sep 09 15:25:35.165008 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.164768    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4637ce97-9134-469e-9b3c-388508a3858b-trusted-ca\") pod \"4637ce97-9134-469e-9b3c-388508a3858b\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") "
Sep 09 15:25:35.165008 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.164808    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fgfs\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-kube-api-access-9fgfs\") pod \"4637ce97-9134-469e-9b3c-388508a3858b\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") "
Sep 09 15:25:35.165008 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.164843    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-bound-sa-token\") pod \"4637ce97-9134-469e-9b3c-388508a3858b\" (UID: \"4637ce97-9134-469e-9b3c-388508a3858b\") "
Sep 09 15:25:35.165259 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.165229    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4637ce97-9134-469e-9b3c-388508a3858b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4637ce97-9134-469e-9b3c-388508a3858b" (UID: "4637ce97-9134-469e-9b3c-388508a3858b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:25:35.165721 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.165646    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4637ce97-9134-469e-9b3c-388508a3858b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4637ce97-9134-469e-9b3c-388508a3858b" (UID: "4637ce97-9134-469e-9b3c-388508a3858b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:25:35.167332 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.167290    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4637ce97-9134-469e-9b3c-388508a3858b" (UID: "4637ce97-9134-469e-9b3c-388508a3858b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:25:35.167438 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.167324    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4637ce97-9134-469e-9b3c-388508a3858b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4637ce97-9134-469e-9b3c-388508a3858b" (UID: "4637ce97-9134-469e-9b3c-388508a3858b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:25:35.167577 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.167559    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4637ce97-9134-469e-9b3c-388508a3858b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4637ce97-9134-469e-9b3c-388508a3858b" (UID: "4637ce97-9134-469e-9b3c-388508a3858b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:25:35.167654 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.167633    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4637ce97-9134-469e-9b3c-388508a3858b" (UID: "4637ce97-9134-469e-9b3c-388508a3858b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:25:35.167709 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.167694    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-kube-api-access-9fgfs" (OuterVolumeSpecName: "kube-api-access-9fgfs") pod "4637ce97-9134-469e-9b3c-388508a3858b" (UID: "4637ce97-9134-469e-9b3c-388508a3858b"). InnerVolumeSpecName "kube-api-access-9fgfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:25:35.173681 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.173662    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4637ce97-9134-469e-9b3c-388508a3858b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4637ce97-9134-469e-9b3c-388508a3858b" (UID: "4637ce97-9134-469e-9b3c-388508a3858b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:25:35.266174 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.266148    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4637ce97-9134-469e-9b3c-388508a3858b-trusted-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:25:35.266174 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.266171    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fgfs\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-kube-api-access-9fgfs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:25:35.266313 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.266180    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-bound-sa-token\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:25:35.266313 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.266189    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4637ce97-9134-469e-9b3c-388508a3858b-registry-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:25:35.266313 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.266197    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4637ce97-9134-469e-9b3c-388508a3858b-ca-trust-extracted\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:25:35.266313 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.266204    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4637ce97-9134-469e-9b3c-388508a3858b-installation-pull-secrets\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:25:35.266313 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.266213    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4637ce97-9134-469e-9b3c-388508a3858b-image-registry-private-configuration\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:25:35.266313 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.266222    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4637ce97-9134-469e-9b3c-388508a3858b-registry-certificates\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:25:35.338482 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.338423    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:35.963272 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.963244    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq"
Sep 09 15:25:35.963272 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.963256    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8945c7fd4-h4fdq" event={"ID":"4637ce97-9134-469e-9b3c-388508a3858b","Type":"ContainerDied","Data":"d73c67c699885573f409e594d2aab7794182c7c1f9a98d89466fc955b529a6c5"}
Sep 09 15:25:35.963739 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.963315    2578 scope.go:117] "RemoveContainer" containerID="52826c00b9a2eca2d330eb74b0206d62da23d822d8bac7a583971d16120d713a"
Sep 09 15:25:35.983652 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.983626    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8945c7fd4-h4fdq"]
Sep 09 15:25:35.988896 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:35.988875    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8945c7fd4-h4fdq"]
Sep 09 15:25:36.865573 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:36.865541    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4637ce97-9134-469e-9b3c-388508a3858b" path="/var/lib/kubelet/pods/4637ce97-9134-469e-9b3c-388508a3858b/volumes"
Sep 09 15:25:40.973862 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:40.973812    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:41.076907 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:41.076872    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:25:59.115771 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:59.115737    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:25:59.180567 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:25:59.180540    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:00.092752 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:00.092726    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:11.793618 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:11.793582    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:26:12.033999 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:12.033953    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:26:12.812066 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:12.812029    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 15:26:12.812473 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:12.812350    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="alertmanager" containerID="cri-o://f3e70461f78a4a2974285c84818f6a67da6ce0ddb21940c5e92759aa7ea1651a" gracePeriod=120
Sep 09 15:26:12.812473 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:12.812444    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="kube-rbac-proxy-web" containerID="cri-o://256821f3e6878e29a06e5422441e60914359a22350ce928876d47ff7f78ae338" gracePeriod=120
Sep 09 15:26:12.812586 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:12.812473    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="prom-label-proxy" containerID="cri-o://986dcc74e7d382a7d9eca69752667fa3e27c301a30c0fa56ba48b643f3231a78" gracePeriod=120
Sep 09 15:26:12.812586 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:12.812486    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="config-reloader" containerID="cri-o://5665056cff09d3a3f51f2a9ea13dca17fc14ea85adc815b494db807c02daec4e" gracePeriod=120
Sep 09 15:26:12.812750 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:12.812581    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="kube-rbac-proxy" containerID="cri-o://c3382a8104ee11b63c7169dbc13fc88bbae4531ed5d1ecb52b71e851d54004a3" gracePeriod=120
Sep 09 15:26:12.812750 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:12.812424    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="kube-rbac-proxy-metric" containerID="cri-o://9e217a1d7fac1a7ea777a35cd446a55b22fa8f672f05458c292f9bb0aec708c5" gracePeriod=120
Sep 09 15:26:13.127531 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:13.127451    2578 generic.go:356] "Generic (PLEG): container finished" podID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerID="986dcc74e7d382a7d9eca69752667fa3e27c301a30c0fa56ba48b643f3231a78" exitCode=0
Sep 09 15:26:13.127531 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:13.127475    2578 generic.go:356] "Generic (PLEG): container finished" podID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerID="9e217a1d7fac1a7ea777a35cd446a55b22fa8f672f05458c292f9bb0aec708c5" exitCode=0
Sep 09 15:26:13.127531 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:13.127480    2578 generic.go:356] "Generic (PLEG): container finished" podID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerID="c3382a8104ee11b63c7169dbc13fc88bbae4531ed5d1ecb52b71e851d54004a3" exitCode=0
Sep 09 15:26:13.127531 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:13.127486    2578 generic.go:356] "Generic (PLEG): container finished" podID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerID="5665056cff09d3a3f51f2a9ea13dca17fc14ea85adc815b494db807c02daec4e" exitCode=0
Sep 09 15:26:13.127531 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:13.127490    2578 generic.go:356] "Generic (PLEG): container finished" podID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerID="f3e70461f78a4a2974285c84818f6a67da6ce0ddb21940c5e92759aa7ea1651a" exitCode=0
Sep 09 15:26:13.127531 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:13.127523    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6686e44e-ff16-458c-ad0c-3b65eec595e1","Type":"ContainerDied","Data":"986dcc74e7d382a7d9eca69752667fa3e27c301a30c0fa56ba48b643f3231a78"}
Sep 09 15:26:13.127814 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:13.127593    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6686e44e-ff16-458c-ad0c-3b65eec595e1","Type":"ContainerDied","Data":"9e217a1d7fac1a7ea777a35cd446a55b22fa8f672f05458c292f9bb0aec708c5"}
Sep 09 15:26:13.127814 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:13.127620    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6686e44e-ff16-458c-ad0c-3b65eec595e1","Type":"ContainerDied","Data":"c3382a8104ee11b63c7169dbc13fc88bbae4531ed5d1ecb52b71e851d54004a3"}
Sep 09 15:26:13.127814 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:13.127644    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6686e44e-ff16-458c-ad0c-3b65eec595e1","Type":"ContainerDied","Data":"5665056cff09d3a3f51f2a9ea13dca17fc14ea85adc815b494db807c02daec4e"}
Sep 09 15:26:13.127814 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:13.127665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6686e44e-ff16-458c-ad0c-3b65eec595e1","Type":"ContainerDied","Data":"f3e70461f78a4a2974285c84818f6a67da6ce0ddb21940c5e92759aa7ea1651a"}
Sep 09 15:26:14.056375 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.056353    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.101181 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.101147    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6686e44e-ff16-458c-ad0c-3b65eec595e1-config-out\") pod \"6686e44e-ff16-458c-ad0c-3b65eec595e1\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") "
Sep 09 15:26:14.101339 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.101205    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6686e44e-ff16-458c-ad0c-3b65eec595e1-metrics-client-ca\") pod \"6686e44e-ff16-458c-ad0c-3b65eec595e1\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") "
Sep 09 15:26:14.101339 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.101256    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-cluster-tls-config\") pod \"6686e44e-ff16-458c-ad0c-3b65eec595e1\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") "
Sep 09 15:26:14.101339 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.101283    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6686e44e-ff16-458c-ad0c-3b65eec595e1-alertmanager-trusted-ca-bundle\") pod \"6686e44e-ff16-458c-ad0c-3b65eec595e1\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") "
Sep 09 15:26:14.101339 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.101309    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-main-tls\") pod \"6686e44e-ff16-458c-ad0c-3b65eec595e1\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") "
Sep 09 15:26:14.101554 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.101370    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-config-volume\") pod \"6686e44e-ff16-458c-ad0c-3b65eec595e1\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") "
Sep 09 15:26:14.101554 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.101411    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-kube-rbac-proxy\") pod \"6686e44e-ff16-458c-ad0c-3b65eec595e1\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") "
Sep 09 15:26:14.101554 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.101439    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6686e44e-ff16-458c-ad0c-3b65eec595e1-alertmanager-main-db\") pod \"6686e44e-ff16-458c-ad0c-3b65eec595e1\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") "
Sep 09 15:26:14.101554 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.101467    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-web-config\") pod \"6686e44e-ff16-458c-ad0c-3b65eec595e1\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") "
Sep 09 15:26:14.101554 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.101502    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"6686e44e-ff16-458c-ad0c-3b65eec595e1\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") "
Sep 09 15:26:14.101554 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.101531    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6686e44e-ff16-458c-ad0c-3b65eec595e1-tls-assets\") pod \"6686e44e-ff16-458c-ad0c-3b65eec595e1\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") "
Sep 09 15:26:14.102035 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.101566    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6686e44e-ff16-458c-ad0c-3b65eec595e1\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") "
Sep 09 15:26:14.102035 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.101601    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rct5p\" (UniqueName: \"kubernetes.io/projected/6686e44e-ff16-458c-ad0c-3b65eec595e1-kube-api-access-rct5p\") pod \"6686e44e-ff16-458c-ad0c-3b65eec595e1\" (UID: \"6686e44e-ff16-458c-ad0c-3b65eec595e1\") "
Sep 09 15:26:14.102035 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.101619    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6686e44e-ff16-458c-ad0c-3b65eec595e1-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6686e44e-ff16-458c-ad0c-3b65eec595e1" (UID: "6686e44e-ff16-458c-ad0c-3b65eec595e1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:26:14.102035 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.101817    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6686e44e-ff16-458c-ad0c-3b65eec595e1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:14.102035 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.101869    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6686e44e-ff16-458c-ad0c-3b65eec595e1-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6686e44e-ff16-458c-ad0c-3b65eec595e1" (UID: "6686e44e-ff16-458c-ad0c-3b65eec595e1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:26:14.102680 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.102647    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6686e44e-ff16-458c-ad0c-3b65eec595e1-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6686e44e-ff16-458c-ad0c-3b65eec595e1" (UID: "6686e44e-ff16-458c-ad0c-3b65eec595e1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:26:14.105222 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.105184    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6686e44e-ff16-458c-ad0c-3b65eec595e1" (UID: "6686e44e-ff16-458c-ad0c-3b65eec595e1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:26:14.105413 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.105383    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6686e44e-ff16-458c-ad0c-3b65eec595e1-config-out" (OuterVolumeSpecName: "config-out") pod "6686e44e-ff16-458c-ad0c-3b65eec595e1" (UID: "6686e44e-ff16-458c-ad0c-3b65eec595e1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:26:14.105557 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.105526    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-config-volume" (OuterVolumeSpecName: "config-volume") pod "6686e44e-ff16-458c-ad0c-3b65eec595e1" (UID: "6686e44e-ff16-458c-ad0c-3b65eec595e1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:26:14.105700 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.105549    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6686e44e-ff16-458c-ad0c-3b65eec595e1" (UID: "6686e44e-ff16-458c-ad0c-3b65eec595e1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:26:14.105940 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.105906    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6686e44e-ff16-458c-ad0c-3b65eec595e1-kube-api-access-rct5p" (OuterVolumeSpecName: "kube-api-access-rct5p") pod "6686e44e-ff16-458c-ad0c-3b65eec595e1" (UID: "6686e44e-ff16-458c-ad0c-3b65eec595e1"). InnerVolumeSpecName "kube-api-access-rct5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:26:14.106058 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.106007    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6686e44e-ff16-458c-ad0c-3b65eec595e1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6686e44e-ff16-458c-ad0c-3b65eec595e1" (UID: "6686e44e-ff16-458c-ad0c-3b65eec595e1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:26:14.107261 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.107240    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6686e44e-ff16-458c-ad0c-3b65eec595e1" (UID: "6686e44e-ff16-458c-ad0c-3b65eec595e1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:26:14.107847 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.107823    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6686e44e-ff16-458c-ad0c-3b65eec595e1" (UID: "6686e44e-ff16-458c-ad0c-3b65eec595e1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:26:14.109670 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.109645    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6686e44e-ff16-458c-ad0c-3b65eec595e1" (UID: "6686e44e-ff16-458c-ad0c-3b65eec595e1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:26:14.118045 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.117994    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-web-config" (OuterVolumeSpecName: "web-config") pod "6686e44e-ff16-458c-ad0c-3b65eec595e1" (UID: "6686e44e-ff16-458c-ad0c-3b65eec595e1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:26:14.135933 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.135904    2578 generic.go:356] "Generic (PLEG): container finished" podID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerID="256821f3e6878e29a06e5422441e60914359a22350ce928876d47ff7f78ae338" exitCode=0
Sep 09 15:26:14.136127 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.135958    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6686e44e-ff16-458c-ad0c-3b65eec595e1","Type":"ContainerDied","Data":"256821f3e6878e29a06e5422441e60914359a22350ce928876d47ff7f78ae338"}
Sep 09 15:26:14.136127 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.136006    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.136127 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.136064    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6686e44e-ff16-458c-ad0c-3b65eec595e1","Type":"ContainerDied","Data":"9a8f3fd1c5aa51f919d0e26c978f6b11f196eb920eb62b645da87896c58b030a"}
Sep 09 15:26:14.136127 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.136113    2578 scope.go:117] "RemoveContainer" containerID="986dcc74e7d382a7d9eca69752667fa3e27c301a30c0fa56ba48b643f3231a78"
Sep 09 15:26:14.143972 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.143924    2578 scope.go:117] "RemoveContainer" containerID="9e217a1d7fac1a7ea777a35cd446a55b22fa8f672f05458c292f9bb0aec708c5"
Sep 09 15:26:14.151350 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.151335    2578 scope.go:117] "RemoveContainer" containerID="c3382a8104ee11b63c7169dbc13fc88bbae4531ed5d1ecb52b71e851d54004a3"
Sep 09 15:26:14.158579 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.158045    2578 scope.go:117] "RemoveContainer" containerID="256821f3e6878e29a06e5422441e60914359a22350ce928876d47ff7f78ae338"
Sep 09 15:26:14.160765 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.160748    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 15:26:14.163901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.163882    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 15:26:14.167973 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.167157    2578 scope.go:117] "RemoveContainer" containerID="5665056cff09d3a3f51f2a9ea13dca17fc14ea85adc815b494db807c02daec4e"
Sep 09 15:26:14.174429 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.174412    2578 scope.go:117] "RemoveContainer" containerID="f3e70461f78a4a2974285c84818f6a67da6ce0ddb21940c5e92759aa7ea1651a"
Sep 09 15:26:14.181515 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.181500    2578 scope.go:117] "RemoveContainer" containerID="e391698c3c4fd10898c1259912bedd5cc03a22b3626837c4738a029601ec4ff7"
Sep 09 15:26:14.188683 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.188665    2578 scope.go:117] "RemoveContainer" containerID="986dcc74e7d382a7d9eca69752667fa3e27c301a30c0fa56ba48b643f3231a78"
Sep 09 15:26:14.188888 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.188867    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 15:26:14.189030 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:26:14.189007    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"986dcc74e7d382a7d9eca69752667fa3e27c301a30c0fa56ba48b643f3231a78\": container with ID starting with 986dcc74e7d382a7d9eca69752667fa3e27c301a30c0fa56ba48b643f3231a78 not found: ID does not exist" containerID="986dcc74e7d382a7d9eca69752667fa3e27c301a30c0fa56ba48b643f3231a78"
Sep 09 15:26:14.189252 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189041    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"986dcc74e7d382a7d9eca69752667fa3e27c301a30c0fa56ba48b643f3231a78"} err="failed to get container status \"986dcc74e7d382a7d9eca69752667fa3e27c301a30c0fa56ba48b643f3231a78\": rpc error: code = NotFound desc = could not find container \"986dcc74e7d382a7d9eca69752667fa3e27c301a30c0fa56ba48b643f3231a78\": container with ID starting with 986dcc74e7d382a7d9eca69752667fa3e27c301a30c0fa56ba48b643f3231a78 not found: ID does not exist"
Sep 09 15:26:14.189252 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189070    2578 scope.go:117] "RemoveContainer" containerID="9e217a1d7fac1a7ea777a35cd446a55b22fa8f672f05458c292f9bb0aec708c5"
Sep 09 15:26:14.189252 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189239    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="config-reloader"
Sep 09 15:26:14.189252 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189253    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="config-reloader"
Sep 09 15:26:14.189432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189264    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="prom-label-proxy"
Sep 09 15:26:14.189432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189270    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="prom-label-proxy"
Sep 09 15:26:14.189432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189278    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="alertmanager"
Sep 09 15:26:14.189432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189287    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="alertmanager"
Sep 09 15:26:14.189432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189293    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="kube-rbac-proxy-web"
Sep 09 15:26:14.189432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189298    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="kube-rbac-proxy-web"
Sep 09 15:26:14.189432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189304    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="kube-rbac-proxy-metric"
Sep 09 15:26:14.189432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189309    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="kube-rbac-proxy-metric"
Sep 09 15:26:14.189432 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:26:14.189312    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e217a1d7fac1a7ea777a35cd446a55b22fa8f672f05458c292f9bb0aec708c5\": container with ID starting with 9e217a1d7fac1a7ea777a35cd446a55b22fa8f672f05458c292f9bb0aec708c5 not found: ID does not exist" containerID="9e217a1d7fac1a7ea777a35cd446a55b22fa8f672f05458c292f9bb0aec708c5"
Sep 09 15:26:14.189432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189333    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e217a1d7fac1a7ea777a35cd446a55b22fa8f672f05458c292f9bb0aec708c5"} err="failed to get container status \"9e217a1d7fac1a7ea777a35cd446a55b22fa8f672f05458c292f9bb0aec708c5\": rpc error: code = NotFound desc = could not find container \"9e217a1d7fac1a7ea777a35cd446a55b22fa8f672f05458c292f9bb0aec708c5\": container with ID starting with 9e217a1d7fac1a7ea777a35cd446a55b22fa8f672f05458c292f9bb0aec708c5 not found: ID does not exist"
Sep 09 15:26:14.189432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189347    2578 scope.go:117] "RemoveContainer" containerID="c3382a8104ee11b63c7169dbc13fc88bbae4531ed5d1ecb52b71e851d54004a3"
Sep 09 15:26:14.189432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189317    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="init-config-reloader"
Sep 09 15:26:14.189432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189388    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="init-config-reloader"
Sep 09 15:26:14.189432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189406    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4637ce97-9134-469e-9b3c-388508a3858b" containerName="registry"
Sep 09 15:26:14.189432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189412    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4637ce97-9134-469e-9b3c-388508a3858b" containerName="registry"
Sep 09 15:26:14.189432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189435    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="kube-rbac-proxy"
Sep 09 15:26:14.189432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189441    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="kube-rbac-proxy"
Sep 09 15:26:14.190285 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189533    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="kube-rbac-proxy"
Sep 09 15:26:14.190285 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189548    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="alertmanager"
Sep 09 15:26:14.190285 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189555    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="kube-rbac-proxy-web"
Sep 09 15:26:14.190285 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189561    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="prom-label-proxy"
Sep 09 15:26:14.190285 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189566    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="kube-rbac-proxy-metric"
Sep 09 15:26:14.190285 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189571    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" containerName="config-reloader"
Sep 09 15:26:14.190285 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189577    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4637ce97-9134-469e-9b3c-388508a3858b" containerName="registry"
Sep 09 15:26:14.190285 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:26:14.189592    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3382a8104ee11b63c7169dbc13fc88bbae4531ed5d1ecb52b71e851d54004a3\": container with ID starting with c3382a8104ee11b63c7169dbc13fc88bbae4531ed5d1ecb52b71e851d54004a3 not found: ID does not exist" containerID="c3382a8104ee11b63c7169dbc13fc88bbae4531ed5d1ecb52b71e851d54004a3"
Sep 09 15:26:14.190285 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189614    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3382a8104ee11b63c7169dbc13fc88bbae4531ed5d1ecb52b71e851d54004a3"} err="failed to get container status \"c3382a8104ee11b63c7169dbc13fc88bbae4531ed5d1ecb52b71e851d54004a3\": rpc error: code = NotFound desc = could not find container \"c3382a8104ee11b63c7169dbc13fc88bbae4531ed5d1ecb52b71e851d54004a3\": container with ID starting with c3382a8104ee11b63c7169dbc13fc88bbae4531ed5d1ecb52b71e851d54004a3 not found: ID does not exist"
Sep 09 15:26:14.190285 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189629    2578 scope.go:117] "RemoveContainer" containerID="256821f3e6878e29a06e5422441e60914359a22350ce928876d47ff7f78ae338"
Sep 09 15:26:14.190285 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:26:14.189948    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"256821f3e6878e29a06e5422441e60914359a22350ce928876d47ff7f78ae338\": container with ID starting with 256821f3e6878e29a06e5422441e60914359a22350ce928876d47ff7f78ae338 not found: ID does not exist" containerID="256821f3e6878e29a06e5422441e60914359a22350ce928876d47ff7f78ae338"
Sep 09 15:26:14.190285 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.189997    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"256821f3e6878e29a06e5422441e60914359a22350ce928876d47ff7f78ae338"} err="failed to get container status \"256821f3e6878e29a06e5422441e60914359a22350ce928876d47ff7f78ae338\": rpc error: code = NotFound desc = could not find container \"256821f3e6878e29a06e5422441e60914359a22350ce928876d47ff7f78ae338\": container with ID starting with 256821f3e6878e29a06e5422441e60914359a22350ce928876d47ff7f78ae338 not found: ID does not exist"
Sep 09 15:26:14.190285 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.190018    2578 scope.go:117] "RemoveContainer" containerID="5665056cff09d3a3f51f2a9ea13dca17fc14ea85adc815b494db807c02daec4e"
Sep 09 15:26:14.190804 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:26:14.190283    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5665056cff09d3a3f51f2a9ea13dca17fc14ea85adc815b494db807c02daec4e\": container with ID starting with 5665056cff09d3a3f51f2a9ea13dca17fc14ea85adc815b494db807c02daec4e not found: ID does not exist" containerID="5665056cff09d3a3f51f2a9ea13dca17fc14ea85adc815b494db807c02daec4e"
Sep 09 15:26:14.190804 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.190312    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5665056cff09d3a3f51f2a9ea13dca17fc14ea85adc815b494db807c02daec4e"} err="failed to get container status \"5665056cff09d3a3f51f2a9ea13dca17fc14ea85adc815b494db807c02daec4e\": rpc error: code = NotFound desc = could not find container \"5665056cff09d3a3f51f2a9ea13dca17fc14ea85adc815b494db807c02daec4e\": container with ID starting with 5665056cff09d3a3f51f2a9ea13dca17fc14ea85adc815b494db807c02daec4e not found: ID does not exist"
Sep 09 15:26:14.190804 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.190332    2578 scope.go:117] "RemoveContainer" containerID="f3e70461f78a4a2974285c84818f6a67da6ce0ddb21940c5e92759aa7ea1651a"
Sep 09 15:26:14.190804 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:26:14.190600    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3e70461f78a4a2974285c84818f6a67da6ce0ddb21940c5e92759aa7ea1651a\": container with ID starting with f3e70461f78a4a2974285c84818f6a67da6ce0ddb21940c5e92759aa7ea1651a not found: ID does not exist" containerID="f3e70461f78a4a2974285c84818f6a67da6ce0ddb21940c5e92759aa7ea1651a"
Sep 09 15:26:14.190804 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.190635    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3e70461f78a4a2974285c84818f6a67da6ce0ddb21940c5e92759aa7ea1651a"} err="failed to get container status \"f3e70461f78a4a2974285c84818f6a67da6ce0ddb21940c5e92759aa7ea1651a\": rpc error: code = NotFound desc = could not find container \"f3e70461f78a4a2974285c84818f6a67da6ce0ddb21940c5e92759aa7ea1651a\": container with ID starting with f3e70461f78a4a2974285c84818f6a67da6ce0ddb21940c5e92759aa7ea1651a not found: ID does not exist"
Sep 09 15:26:14.190804 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.190651    2578 scope.go:117] "RemoveContainer" containerID="e391698c3c4fd10898c1259912bedd5cc03a22b3626837c4738a029601ec4ff7"
Sep 09 15:26:14.191038 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:26:14.190906    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e391698c3c4fd10898c1259912bedd5cc03a22b3626837c4738a029601ec4ff7\": container with ID starting with e391698c3c4fd10898c1259912bedd5cc03a22b3626837c4738a029601ec4ff7 not found: ID does not exist" containerID="e391698c3c4fd10898c1259912bedd5cc03a22b3626837c4738a029601ec4ff7"
Sep 09 15:26:14.191038 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.190921    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e391698c3c4fd10898c1259912bedd5cc03a22b3626837c4738a029601ec4ff7"} err="failed to get container status \"e391698c3c4fd10898c1259912bedd5cc03a22b3626837c4738a029601ec4ff7\": rpc error: code = NotFound desc = could not find container \"e391698c3c4fd10898c1259912bedd5cc03a22b3626837c4738a029601ec4ff7\": container with ID starting with e391698c3c4fd10898c1259912bedd5cc03a22b3626837c4738a029601ec4ff7 not found: ID does not exist"
Sep 09 15:26:14.192500 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.192486    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.194640 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.194622    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 15:26:14.194743 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.194680    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 15:26:14.194743 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.194682    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 15:26:14.194945 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.194929    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8gt4p\""
Sep 09 15:26:14.195226 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.195197    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 15:26:14.195317 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.195256    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 15:26:14.195386 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.195367    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 15:26:14.195444 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.195397    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 15:26:14.195776 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.195735    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 15:26:14.203217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.201995    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 15:26:14.203217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.202796    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-web-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:14.203217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.202816    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:14.203217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.202832    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6686e44e-ff16-458c-ad0c-3b65eec595e1-tls-assets\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:14.203217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.202847    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:14.203217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.202876    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rct5p\" (UniqueName: \"kubernetes.io/projected/6686e44e-ff16-458c-ad0c-3b65eec595e1-kube-api-access-rct5p\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:14.203217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.202891    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6686e44e-ff16-458c-ad0c-3b65eec595e1-config-out\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:14.203217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.202903    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6686e44e-ff16-458c-ad0c-3b65eec595e1-metrics-client-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:14.203217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.202915    2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-cluster-tls-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:14.203217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.202933    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-main-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:14.203217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.202946    2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-config-volume\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:14.203217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.202958    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6686e44e-ff16-458c-ad0c-3b65eec595e1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:14.203217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.202971    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6686e44e-ff16-458c-ad0c-3b65eec595e1-alertmanager-main-db\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:14.206937 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.206916    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 15:26:14.303759 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.303721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-web-config\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.303759 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.303766    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.303958 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.303784    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.303958 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.303843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d053f8b6-bff5-4174-acd4-8ea1a06f9192-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.303958 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.303878    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.303958 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.303895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqxw7\" (UniqueName: \"kubernetes.io/projected/d053f8b6-bff5-4174-acd4-8ea1a06f9192-kube-api-access-nqxw7\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.303958 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.303920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-config-volume\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.303958 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.303935    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.304187 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.303975    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d053f8b6-bff5-4174-acd4-8ea1a06f9192-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.304187 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.304068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d053f8b6-bff5-4174-acd4-8ea1a06f9192-config-out\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.304187 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.304100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d053f8b6-bff5-4174-acd4-8ea1a06f9192-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.304187 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.304118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.304187 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.304148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d053f8b6-bff5-4174-acd4-8ea1a06f9192-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.405109 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.405029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d053f8b6-bff5-4174-acd4-8ea1a06f9192-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.405109 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.405068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.405109 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.405096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d053f8b6-bff5-4174-acd4-8ea1a06f9192-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.405316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.405159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-web-config\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.405359 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.405303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.405359 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.405341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.405464 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.405369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d053f8b6-bff5-4174-acd4-8ea1a06f9192-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.405464 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.405396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.405800 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.405765    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d053f8b6-bff5-4174-acd4-8ea1a06f9192-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.406005 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.405963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d053f8b6-bff5-4174-acd4-8ea1a06f9192-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.406084 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.406059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxw7\" (UniqueName: \"kubernetes.io/projected/d053f8b6-bff5-4174-acd4-8ea1a06f9192-kube-api-access-nqxw7\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.406142 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.406121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-config-volume\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.406190 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.406150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.406190 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.406185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d053f8b6-bff5-4174-acd4-8ea1a06f9192-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.406285 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.406220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d053f8b6-bff5-4174-acd4-8ea1a06f9192-config-out\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.408681 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.408442    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.408681 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.408537    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.408681 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.408613    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.408681 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.408657    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.408932 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.408696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-web-config\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.408932 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.408874    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d053f8b6-bff5-4174-acd4-8ea1a06f9192-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.409035 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.409021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d053f8b6-bff5-4174-acd4-8ea1a06f9192-config-out\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.409168 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.409151    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.409363 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.409347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d053f8b6-bff5-4174-acd4-8ea1a06f9192-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.410435 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.410419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d053f8b6-bff5-4174-acd4-8ea1a06f9192-config-volume\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.413146 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.413127    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqxw7\" (UniqueName: \"kubernetes.io/projected/d053f8b6-bff5-4174-acd4-8ea1a06f9192-kube-api-access-nqxw7\") pod \"alertmanager-main-0\" (UID: \"d053f8b6-bff5-4174-acd4-8ea1a06f9192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.507685 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.507653    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 15:26:14.781461 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.781436    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 15:26:14.783415 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:26:14.783386    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd053f8b6_bff5_4174_acd4_8ea1a06f9192.slice/crio-fff597d3e07a4cda1110c79264bd5e20d842ddc31bcd1e1f7bd3d17daa8ddf53 WatchSource:0}: Error finding container fff597d3e07a4cda1110c79264bd5e20d842ddc31bcd1e1f7bd3d17daa8ddf53: Status 404 returned error can't find the container with id fff597d3e07a4cda1110c79264bd5e20d842ddc31bcd1e1f7bd3d17daa8ddf53
Sep 09 15:26:14.865351 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:14.865326    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6686e44e-ff16-458c-ad0c-3b65eec595e1" path="/var/lib/kubelet/pods/6686e44e-ff16-458c-ad0c-3b65eec595e1/volumes"
Sep 09 15:26:15.141278 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:15.141246    2578 generic.go:356] "Generic (PLEG): container finished" podID="d053f8b6-bff5-4174-acd4-8ea1a06f9192" containerID="7e616f8fe77e94271bdae84ffd21ceeff8ec117585b33ffbec4b27e05585f8a2" exitCode=0
Sep 09 15:26:15.141698 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:15.141335    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d053f8b6-bff5-4174-acd4-8ea1a06f9192","Type":"ContainerDied","Data":"7e616f8fe77e94271bdae84ffd21ceeff8ec117585b33ffbec4b27e05585f8a2"}
Sep 09 15:26:15.141698 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:15.141413    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d053f8b6-bff5-4174-acd4-8ea1a06f9192","Type":"ContainerStarted","Data":"fff597d3e07a4cda1110c79264bd5e20d842ddc31bcd1e1f7bd3d17daa8ddf53"}
Sep 09 15:26:16.150457 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.150418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d053f8b6-bff5-4174-acd4-8ea1a06f9192","Type":"ContainerStarted","Data":"13190e6a8399469b787144c3632630d97270ea20adfdaafff605da088bbf40e0"}
Sep 09 15:26:16.151031 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.150488    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d053f8b6-bff5-4174-acd4-8ea1a06f9192","Type":"ContainerStarted","Data":"da77927a66673acb7a22e9a443640ff13ac6e4aaa9bcb5f9faa6f271825a60ae"}
Sep 09 15:26:16.151031 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.150519    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d053f8b6-bff5-4174-acd4-8ea1a06f9192","Type":"ContainerStarted","Data":"891d27e7afd83445e2ca5bb2dd9b8a74032e27cca45d8b4957181ca46026872e"}
Sep 09 15:26:16.151031 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.150545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d053f8b6-bff5-4174-acd4-8ea1a06f9192","Type":"ContainerStarted","Data":"cc3840313293d66d09c298e1252960e027a789773f863a5ae0739726b52bcc8a"}
Sep 09 15:26:16.151031 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.150569    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d053f8b6-bff5-4174-acd4-8ea1a06f9192","Type":"ContainerStarted","Data":"27837068f78cdfd5cffba726f884b307716aceb2626fca8658031d326c83d1e4"}
Sep 09 15:26:16.151031 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.150591    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d053f8b6-bff5-4174-acd4-8ea1a06f9192","Type":"ContainerStarted","Data":"eb1f11cf517fc34b02db09f4b41e968f9e86170e71557d2c85c0a932eed0bb71"}
Sep 09 15:26:16.175316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.175265    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.160014939 podStartE2EDuration="2.175248634s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.233695ms" isStatefulPod=false podCreationTimestamp="2026-09-09 15:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:26:16.173102195 +0000 UTC m=+365.815144209" watchObservedRunningTime="2026-09-09 15:26:16.175248634 +0000 UTC m=+365.817290673"
Sep 09 15:26:16.869861 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.869821    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"]
Sep 09 15:26:16.872525 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.872504    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:16.875146 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.875121    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 09 15:26:16.875272 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.875158    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 09 15:26:16.875380 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.875360    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 09 15:26:16.875462 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.875446    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 09 15:26:16.875525 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.875474    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-486st\""
Sep 09 15:26:16.875729 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.875714    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 09 15:26:16.884357 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.884163    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 09 15:26:16.891565 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.889724    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"]
Sep 09 15:26:16.930377 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.930342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1981bf38-60c0-4674-bb1e-cf6fc10eabff-serving-certs-ca-bundle\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:16.930567 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.930391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1981bf38-60c0-4674-bb1e-cf6fc10eabff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:16.930567 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.930424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1981bf38-60c0-4674-bb1e-cf6fc10eabff-metrics-client-ca\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:16.930567 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.930467    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mv7v\" (UniqueName: \"kubernetes.io/projected/1981bf38-60c0-4674-bb1e-cf6fc10eabff-kube-api-access-2mv7v\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:16.930567 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.930510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1981bf38-60c0-4674-bb1e-cf6fc10eabff-secret-telemeter-client\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:16.930777 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.930603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1981bf38-60c0-4674-bb1e-cf6fc10eabff-telemeter-client-tls\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:16.930777 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.930625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1981bf38-60c0-4674-bb1e-cf6fc10eabff-federate-client-tls\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:16.930777 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:16.930679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1981bf38-60c0-4674-bb1e-cf6fc10eabff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:17.031384 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.031350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1981bf38-60c0-4674-bb1e-cf6fc10eabff-telemeter-client-tls\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:17.031384 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.031397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1981bf38-60c0-4674-bb1e-cf6fc10eabff-federate-client-tls\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:17.031654 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.031421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1981bf38-60c0-4674-bb1e-cf6fc10eabff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:17.031654 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.031445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1981bf38-60c0-4674-bb1e-cf6fc10eabff-serving-certs-ca-bundle\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:17.031654 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.031467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1981bf38-60c0-4674-bb1e-cf6fc10eabff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:17.031654 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.031501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1981bf38-60c0-4674-bb1e-cf6fc10eabff-metrics-client-ca\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:17.031654 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.031540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mv7v\" (UniqueName: \"kubernetes.io/projected/1981bf38-60c0-4674-bb1e-cf6fc10eabff-kube-api-access-2mv7v\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:17.031654 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.031583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1981bf38-60c0-4674-bb1e-cf6fc10eabff-secret-telemeter-client\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:17.032424 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.032343    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1981bf38-60c0-4674-bb1e-cf6fc10eabff-metrics-client-ca\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:17.032563 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.032474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1981bf38-60c0-4674-bb1e-cf6fc10eabff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:17.032563 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.032398    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1981bf38-60c0-4674-bb1e-cf6fc10eabff-serving-certs-ca-bundle\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:17.034340 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.034304    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1981bf38-60c0-4674-bb1e-cf6fc10eabff-federate-client-tls\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:17.034460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.034404    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1981bf38-60c0-4674-bb1e-cf6fc10eabff-secret-telemeter-client\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:17.034527 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.034469    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1981bf38-60c0-4674-bb1e-cf6fc10eabff-telemeter-client-tls\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:17.034688 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.034669    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1981bf38-60c0-4674-bb1e-cf6fc10eabff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:17.039915 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.039886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mv7v\" (UniqueName: \"kubernetes.io/projected/1981bf38-60c0-4674-bb1e-cf6fc10eabff-kube-api-access-2mv7v\") pod \"telemeter-client-797d4dcc75-8z4fk\" (UID: \"1981bf38-60c0-4674-bb1e-cf6fc10eabff\") " pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:17.164084 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.164051    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 15:26:17.164495 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.164347    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="prometheus" containerID="cri-o://fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5" gracePeriod=600
Sep 09 15:26:17.164495 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.164397    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="kube-rbac-proxy" containerID="cri-o://ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce" gracePeriod=600
Sep 09 15:26:17.164495 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.164422    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="config-reloader" containerID="cri-o://cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e" gracePeriod=600
Sep 09 15:26:17.164495 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.164436    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="kube-rbac-proxy-web" containerID="cri-o://70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a" gracePeriod=600
Sep 09 15:26:17.164705 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.164405    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="thanos-sidecar" containerID="cri-o://c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df" gracePeriod=600
Sep 09 15:26:17.164705 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.164545    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="kube-rbac-proxy-thanos" containerID="cri-o://c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d" gracePeriod=600
Sep 09 15:26:17.184453 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.184430    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"
Sep 09 15:26:17.479741 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.479717    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-797d4dcc75-8z4fk"]
Sep 09 15:26:17.502974 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:26:17.502903    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1981bf38_60c0_4674_bb1e_cf6fc10eabff.slice/crio-6e7ba5368637d6ba0ac5145a633c0f9eb99e382dabaa41d98fe7bfa4f8e5a4fc WatchSource:0}: Error finding container 6e7ba5368637d6ba0ac5145a633c0f9eb99e382dabaa41d98fe7bfa4f8e5a4fc: Status 404 returned error can't find the container with id 6e7ba5368637d6ba0ac5145a633c0f9eb99e382dabaa41d98fe7bfa4f8e5a4fc
Sep 09 15:26:17.505268 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.505250    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 15:26:17.599826 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.599804    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:17.636619 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.636585    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf262\" (UniqueName: \"kubernetes.io/projected/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-kube-api-access-wf262\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.636783 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.636644    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-rulefiles-0\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.636783 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.636704    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-config\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.636783 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.636730    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-kube-rbac-proxy\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.639564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.637173    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-trusted-ca-bundle\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.639564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.637215    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-thanos-prometheus-http-client-file\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.639564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.637245    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.639564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.637275    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-config-out\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.639564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.637344    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-metrics-client-certs\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.639564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.637381    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-configmap-metrics-client-ca\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.639564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.637415    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-rulefiles-2\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.639564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.637469    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-prometheus-k8s-tls\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.639564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.637501    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-grpc-tls\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.639564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.637530    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-configmap-kubelet-serving-ca-bundle\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.639564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.637553    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-web-config\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.639564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.637588    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-db\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.639564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.637625    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.639564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.637658    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-configmap-serving-certs-ca-bundle\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.639564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.637684    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-rulefiles-1\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.639564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.637710    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-tls-assets\") pod \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\" (UID: \"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa\") "
Sep 09 15:26:17.639564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.638476    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:26:17.640968 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.639687    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:26:17.640968 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.640264    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:26:17.640968 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.640356    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:26:17.640968 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.640916    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:26:17.641224 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.641053    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:26:17.641279 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.641239    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-config" (OuterVolumeSpecName: "config") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:26:17.641330 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.641294    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:26:17.641330 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.641291    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:26:17.641449 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.641430    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:26:17.642285 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.642257    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-kube-api-access-wf262" (OuterVolumeSpecName: "kube-api-access-wf262") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "kube-api-access-wf262". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:26:17.642540 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.642502    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:26:17.643898 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.643864    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:26:17.644036 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.643943    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:26:17.644036 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.643949    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:26:17.644371 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.644324    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:26:17.644455 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.644402    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-config-out" (OuterVolumeSpecName: "config-out") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:26:17.644532 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.644498    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:26:17.645198 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.645178    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:26:17.656923 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.656889    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-web-config" (OuterVolumeSpecName: "web-config") pod "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" (UID: "de3e59e7-b0e3-4238-93b6-b0531bd7e3aa"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:26:17.739357 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739263    2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-metrics-client-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739357 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739303    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-configmap-metrics-client-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739357 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739313    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739357 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739322    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739357 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739334    2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-grpc-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739357 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739344    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739357 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739352    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-web-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739357 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739363    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-db\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739357 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739372    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739749 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739380    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739749 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739388    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739749 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739396    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-tls-assets\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739749 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739404    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf262\" (UniqueName: \"kubernetes.io/projected/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-kube-api-access-wf262\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739749 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739412    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739749 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739420    2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-config\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739749 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739427    2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-kube-rbac-proxy\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739749 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739435    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739749 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739443    2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739749 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739453    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:17.739749 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:17.739461    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa-config-out\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:26:18.164059 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.164022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk" event={"ID":"1981bf38-60c0-4674-bb1e-cf6fc10eabff","Type":"ContainerStarted","Data":"6e7ba5368637d6ba0ac5145a633c0f9eb99e382dabaa41d98fe7bfa4f8e5a4fc"}
Sep 09 15:26:18.167126 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.167094    2578 generic.go:356] "Generic (PLEG): container finished" podID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerID="c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d" exitCode=0
Sep 09 15:26:18.167126 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.167120    2578 generic.go:356] "Generic (PLEG): container finished" podID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerID="ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce" exitCode=0
Sep 09 15:26:18.167126 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.167129    2578 generic.go:356] "Generic (PLEG): container finished" podID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerID="70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a" exitCode=0
Sep 09 15:26:18.167521 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.167138    2578 generic.go:356] "Generic (PLEG): container finished" podID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerID="c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df" exitCode=0
Sep 09 15:26:18.167521 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.167146    2578 generic.go:356] "Generic (PLEG): container finished" podID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerID="cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e" exitCode=0
Sep 09 15:26:18.167521 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.167154    2578 generic.go:356] "Generic (PLEG): container finished" podID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerID="fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5" exitCode=0
Sep 09 15:26:18.167521 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.167170    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa","Type":"ContainerDied","Data":"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d"}
Sep 09 15:26:18.167521 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.167187    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.167521 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.167234    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa","Type":"ContainerDied","Data":"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce"}
Sep 09 15:26:18.167521 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.167265    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa","Type":"ContainerDied","Data":"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a"}
Sep 09 15:26:18.167521 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.167290    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa","Type":"ContainerDied","Data":"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df"}
Sep 09 15:26:18.167521 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.167309    2578 scope.go:117] "RemoveContainer" containerID="c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d"
Sep 09 15:26:18.167521 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.167315    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa","Type":"ContainerDied","Data":"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e"}
Sep 09 15:26:18.167521 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.167340    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa","Type":"ContainerDied","Data":"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5"}
Sep 09 15:26:18.167521 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.167362    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de3e59e7-b0e3-4238-93b6-b0531bd7e3aa","Type":"ContainerDied","Data":"96fe5c498555d5052fc6b947701bd69f49189158e54ea11068996cc2ef191ed8"}
Sep 09 15:26:18.175434 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.175304    2578 scope.go:117] "RemoveContainer" containerID="ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce"
Sep 09 15:26:18.182438 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.182417    2578 scope.go:117] "RemoveContainer" containerID="70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a"
Sep 09 15:26:18.189743 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.189725    2578 scope.go:117] "RemoveContainer" containerID="c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df"
Sep 09 15:26:18.190618 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.190598    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 15:26:18.194334 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.194315    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 15:26:18.196736 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.196717    2578 scope.go:117] "RemoveContainer" containerID="cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e"
Sep 09 15:26:18.203754 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.203733    2578 scope.go:117] "RemoveContainer" containerID="fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5"
Sep 09 15:26:18.210799 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.210784    2578 scope.go:117] "RemoveContainer" containerID="f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16"
Sep 09 15:26:18.217621 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.217604    2578 scope.go:117] "RemoveContainer" containerID="c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d"
Sep 09 15:26:18.217892 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:26:18.217869    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d\": container with ID starting with c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d not found: ID does not exist" containerID="c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d"
Sep 09 15:26:18.217929 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.217893    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d"} err="failed to get container status \"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d\": rpc error: code = NotFound desc = could not find container \"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d\": container with ID starting with c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d not found: ID does not exist"
Sep 09 15:26:18.217929 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.217912    2578 scope.go:117] "RemoveContainer" containerID="ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce"
Sep 09 15:26:18.218179 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:26:18.218161    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce\": container with ID starting with ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce not found: ID does not exist" containerID="ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce"
Sep 09 15:26:18.218235 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.218184    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce"} err="failed to get container status \"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce\": rpc error: code = NotFound desc = could not find container \"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce\": container with ID starting with ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce not found: ID does not exist"
Sep 09 15:26:18.218235 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.218200    2578 scope.go:117] "RemoveContainer" containerID="70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a"
Sep 09 15:26:18.218442 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:26:18.218425    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a\": container with ID starting with 70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a not found: ID does not exist" containerID="70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a"
Sep 09 15:26:18.218491 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.218446    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a"} err="failed to get container status \"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a\": rpc error: code = NotFound desc = could not find container \"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a\": container with ID starting with 70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a not found: ID does not exist"
Sep 09 15:26:18.218491 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.218459    2578 scope.go:117] "RemoveContainer" containerID="c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df"
Sep 09 15:26:18.218707 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:26:18.218691    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df\": container with ID starting with c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df not found: ID does not exist" containerID="c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df"
Sep 09 15:26:18.218753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.218709    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df"} err="failed to get container status \"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df\": rpc error: code = NotFound desc = could not find container \"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df\": container with ID starting with c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df not found: ID does not exist"
Sep 09 15:26:18.218753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.218721    2578 scope.go:117] "RemoveContainer" containerID="cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e"
Sep 09 15:26:18.218935 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:26:18.218917    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e\": container with ID starting with cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e not found: ID does not exist" containerID="cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e"
Sep 09 15:26:18.218972 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.218940    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e"} err="failed to get container status \"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e\": rpc error: code = NotFound desc = could not find container \"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e\": container with ID starting with cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e not found: ID does not exist"
Sep 09 15:26:18.218972 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.218953    2578 scope.go:117] "RemoveContainer" containerID="fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5"
Sep 09 15:26:18.219215 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:26:18.219198    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5\": container with ID starting with fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5 not found: ID does not exist" containerID="fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5"
Sep 09 15:26:18.219254 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.219217    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5"} err="failed to get container status \"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5\": rpc error: code = NotFound desc = could not find container \"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5\": container with ID starting with fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5 not found: ID does not exist"
Sep 09 15:26:18.219254 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.219229    2578 scope.go:117] "RemoveContainer" containerID="f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16"
Sep 09 15:26:18.219487 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:26:18.219468    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16\": container with ID starting with f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16 not found: ID does not exist" containerID="f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16"
Sep 09 15:26:18.219539 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.219492    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16"} err="failed to get container status \"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16\": rpc error: code = NotFound desc = could not find container \"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16\": container with ID starting with f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16 not found: ID does not exist"
Sep 09 15:26:18.219539 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.219506    2578 scope.go:117] "RemoveContainer" containerID="c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d"
Sep 09 15:26:18.219739 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.219723    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d"} err="failed to get container status \"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d\": rpc error: code = NotFound desc = could not find container \"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d\": container with ID starting with c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d not found: ID does not exist"
Sep 09 15:26:18.219785 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.219739    2578 scope.go:117] "RemoveContainer" containerID="ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce"
Sep 09 15:26:18.219970 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.219954    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce"} err="failed to get container status \"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce\": rpc error: code = NotFound desc = could not find container \"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce\": container with ID starting with ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce not found: ID does not exist"
Sep 09 15:26:18.220050 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.219970    2578 scope.go:117] "RemoveContainer" containerID="70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a"
Sep 09 15:26:18.220239 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.220221    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a"} err="failed to get container status \"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a\": rpc error: code = NotFound desc = could not find container \"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a\": container with ID starting with 70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a not found: ID does not exist"
Sep 09 15:26:18.220239 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.220240    2578 scope.go:117] "RemoveContainer" containerID="c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df"
Sep 09 15:26:18.220449 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.220432    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df"} err="failed to get container status \"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df\": rpc error: code = NotFound desc = could not find container \"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df\": container with ID starting with c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df not found: ID does not exist"
Sep 09 15:26:18.220488 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.220450    2578 scope.go:117] "RemoveContainer" containerID="cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e"
Sep 09 15:26:18.220656 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.220640    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e"} err="failed to get container status \"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e\": rpc error: code = NotFound desc = could not find container \"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e\": container with ID starting with cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e not found: ID does not exist"
Sep 09 15:26:18.220700 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.220656    2578 scope.go:117] "RemoveContainer" containerID="fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5"
Sep 09 15:26:18.220851 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.220836    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5"} err="failed to get container status \"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5\": rpc error: code = NotFound desc = could not find container \"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5\": container with ID starting with fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5 not found: ID does not exist"
Sep 09 15:26:18.220892 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.220851    2578 scope.go:117] "RemoveContainer" containerID="f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16"
Sep 09 15:26:18.221078 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.221063    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16"} err="failed to get container status \"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16\": rpc error: code = NotFound desc = could not find container \"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16\": container with ID starting with f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16 not found: ID does not exist"
Sep 09 15:26:18.221078 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.221078    2578 scope.go:117] "RemoveContainer" containerID="c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d"
Sep 09 15:26:18.221300 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.221283    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d"} err="failed to get container status \"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d\": rpc error: code = NotFound desc = could not find container \"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d\": container with ID starting with c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d not found: ID does not exist"
Sep 09 15:26:18.221353 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.221299    2578 scope.go:117] "RemoveContainer" containerID="ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce"
Sep 09 15:26:18.221501 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.221486    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce"} err="failed to get container status \"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce\": rpc error: code = NotFound desc = could not find container \"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce\": container with ID starting with ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce not found: ID does not exist"
Sep 09 15:26:18.221501 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.221499    2578 scope.go:117] "RemoveContainer" containerID="70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a"
Sep 09 15:26:18.221749 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.221706    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a"} err="failed to get container status \"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a\": rpc error: code = NotFound desc = could not find container \"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a\": container with ID starting with 70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a not found: ID does not exist"
Sep 09 15:26:18.221828 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.221752    2578 scope.go:117] "RemoveContainer" containerID="c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df"
Sep 09 15:26:18.222109 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.222087    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df"} err="failed to get container status \"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df\": rpc error: code = NotFound desc = could not find container \"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df\": container with ID starting with c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df not found: ID does not exist"
Sep 09 15:26:18.222109 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.222109    2578 scope.go:117] "RemoveContainer" containerID="cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e"
Sep 09 15:26:18.222410 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.222384    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e"} err="failed to get container status \"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e\": rpc error: code = NotFound desc = could not find container \"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e\": container with ID starting with cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e not found: ID does not exist"
Sep 09 15:26:18.222410 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.222411    2578 scope.go:117] "RemoveContainer" containerID="fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5"
Sep 09 15:26:18.222752 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.222735    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5"} err="failed to get container status \"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5\": rpc error: code = NotFound desc = could not find container \"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5\": container with ID starting with fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5 not found: ID does not exist"
Sep 09 15:26:18.222827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.222753    2578 scope.go:117] "RemoveContainer" containerID="f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16"
Sep 09 15:26:18.223104 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.223077    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16"} err="failed to get container status \"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16\": rpc error: code = NotFound desc = could not find container \"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16\": container with ID starting with f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16 not found: ID does not exist"
Sep 09 15:26:18.223163 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.223105    2578 scope.go:117] "RemoveContainer" containerID="c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d"
Sep 09 15:26:18.223370 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.223349    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d"} err="failed to get container status \"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d\": rpc error: code = NotFound desc = could not find container \"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d\": container with ID starting with c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d not found: ID does not exist"
Sep 09 15:26:18.223370 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.223369    2578 scope.go:117] "RemoveContainer" containerID="ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce"
Sep 09 15:26:18.223634 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.223615    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce"} err="failed to get container status \"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce\": rpc error: code = NotFound desc = could not find container \"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce\": container with ID starting with ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce not found: ID does not exist"
Sep 09 15:26:18.223680 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.223637    2578 scope.go:117] "RemoveContainer" containerID="70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a"
Sep 09 15:26:18.223883 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.223864    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a"} err="failed to get container status \"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a\": rpc error: code = NotFound desc = could not find container \"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a\": container with ID starting with 70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a not found: ID does not exist"
Sep 09 15:26:18.223931 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.223885    2578 scope.go:117] "RemoveContainer" containerID="c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df"
Sep 09 15:26:18.224044 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224027    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 15:26:18.224165 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224141    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df"} err="failed to get container status \"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df\": rpc error: code = NotFound desc = could not find container \"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df\": container with ID starting with c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df not found: ID does not exist"
Sep 09 15:26:18.224211 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224167    2578 scope.go:117] "RemoveContainer" containerID="cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e"
Sep 09 15:26:18.224396 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224384    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="kube-rbac-proxy"
Sep 09 15:26:18.224439 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224399    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="kube-rbac-proxy"
Sep 09 15:26:18.224439 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224406    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e"} err="failed to get container status \"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e\": rpc error: code = NotFound desc = could not find container \"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e\": container with ID starting with cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e not found: ID does not exist"
Sep 09 15:26:18.224439 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224429    2578 scope.go:117] "RemoveContainer" containerID="fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5"
Sep 09 15:26:18.224538 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224416    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="kube-rbac-proxy-thanos"
Sep 09 15:26:18.224538 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224477    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="kube-rbac-proxy-thanos"
Sep 09 15:26:18.224538 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224490    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="init-config-reloader"
Sep 09 15:26:18.224538 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224496    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="init-config-reloader"
Sep 09 15:26:18.224538 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224503    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="prometheus"
Sep 09 15:26:18.224538 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224508    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="prometheus"
Sep 09 15:26:18.224538 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224514    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="kube-rbac-proxy-web"
Sep 09 15:26:18.224538 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224522    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="kube-rbac-proxy-web"
Sep 09 15:26:18.224786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224541    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="config-reloader"
Sep 09 15:26:18.224786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224550    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="config-reloader"
Sep 09 15:26:18.224786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224562    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="thanos-sidecar"
Sep 09 15:26:18.224786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224567    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="thanos-sidecar"
Sep 09 15:26:18.224786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224631    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5"} err="failed to get container status \"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5\": rpc error: code = NotFound desc = could not find container \"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5\": container with ID starting with fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5 not found: ID does not exist"
Sep 09 15:26:18.224786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224644    2578 scope.go:117] "RemoveContainer" containerID="f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16"
Sep 09 15:26:18.224786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224652    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="kube-rbac-proxy-thanos"
Sep 09 15:26:18.224786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224662    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="kube-rbac-proxy"
Sep 09 15:26:18.224786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224671    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="config-reloader"
Sep 09 15:26:18.224786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224682    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="kube-rbac-proxy-web"
Sep 09 15:26:18.224786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224692    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="prometheus"
Sep 09 15:26:18.224786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224701    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" containerName="thanos-sidecar"
Sep 09 15:26:18.225180 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224827    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16"} err="failed to get container status \"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16\": rpc error: code = NotFound desc = could not find container \"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16\": container with ID starting with f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16 not found: ID does not exist"
Sep 09 15:26:18.225180 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.224840    2578 scope.go:117] "RemoveContainer" containerID="c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d"
Sep 09 15:26:18.225180 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.225094    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d"} err="failed to get container status \"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d\": rpc error: code = NotFound desc = could not find container \"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d\": container with ID starting with c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d not found: ID does not exist"
Sep 09 15:26:18.225180 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.225113    2578 scope.go:117] "RemoveContainer" containerID="ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce"
Sep 09 15:26:18.225351 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.225336    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce"} err="failed to get container status \"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce\": rpc error: code = NotFound desc = could not find container \"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce\": container with ID starting with ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce not found: ID does not exist"
Sep 09 15:26:18.225400 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.225352    2578 scope.go:117] "RemoveContainer" containerID="70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a"
Sep 09 15:26:18.225593 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.225576    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a"} err="failed to get container status \"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a\": rpc error: code = NotFound desc = could not find container \"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a\": container with ID starting with 70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a not found: ID does not exist"
Sep 09 15:26:18.225643 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.225593    2578 scope.go:117] "RemoveContainer" containerID="c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df"
Sep 09 15:26:18.225824 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.225806    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df"} err="failed to get container status \"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df\": rpc error: code = NotFound desc = could not find container \"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df\": container with ID starting with c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df not found: ID does not exist"
Sep 09 15:26:18.225869 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.225824    2578 scope.go:117] "RemoveContainer" containerID="cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e"
Sep 09 15:26:18.226062 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.226044    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e"} err="failed to get container status \"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e\": rpc error: code = NotFound desc = could not find container \"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e\": container with ID starting with cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e not found: ID does not exist"
Sep 09 15:26:18.226107 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.226063    2578 scope.go:117] "RemoveContainer" containerID="fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5"
Sep 09 15:26:18.226347 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.226323    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5"} err="failed to get container status \"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5\": rpc error: code = NotFound desc = could not find container \"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5\": container with ID starting with fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5 not found: ID does not exist"
Sep 09 15:26:18.226347 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.226346    2578 scope.go:117] "RemoveContainer" containerID="f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16"
Sep 09 15:26:18.226616 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.226597    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16"} err="failed to get container status \"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16\": rpc error: code = NotFound desc = could not find container \"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16\": container with ID starting with f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16 not found: ID does not exist"
Sep 09 15:26:18.226669 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.226619    2578 scope.go:117] "RemoveContainer" containerID="c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d"
Sep 09 15:26:18.226879 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.226854    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d"} err="failed to get container status \"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d\": rpc error: code = NotFound desc = could not find container \"c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d\": container with ID starting with c461c29b0b56b1640531da38ecca5f00544664a356928a1b796a89886dce453d not found: ID does not exist"
Sep 09 15:26:18.226930 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.226879    2578 scope.go:117] "RemoveContainer" containerID="ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce"
Sep 09 15:26:18.227147 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.227131    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce"} err="failed to get container status \"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce\": rpc error: code = NotFound desc = could not find container \"ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce\": container with ID starting with ea33fc4e08e49abf243bb82122b0a4cd4d1af3eacf79757bab04d3c01ac2c8ce not found: ID does not exist"
Sep 09 15:26:18.227197 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.227148    2578 scope.go:117] "RemoveContainer" containerID="70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a"
Sep 09 15:26:18.227512 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.227486    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a"} err="failed to get container status \"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a\": rpc error: code = NotFound desc = could not find container \"70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a\": container with ID starting with 70d00b609c224b45b1532fbe373a20e66b519412525fa47e0a2e8e7d9e1d794a not found: ID does not exist"
Sep 09 15:26:18.227626 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.227518    2578 scope.go:117] "RemoveContainer" containerID="c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df"
Sep 09 15:26:18.227818 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.227801    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df"} err="failed to get container status \"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df\": rpc error: code = NotFound desc = could not find container \"c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df\": container with ID starting with c337c7c9b5ee6f38ca885db02ffd8483c1baa73702ac87c1d256827755f313df not found: ID does not exist"
Sep 09 15:26:18.227884 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.227818    2578 scope.go:117] "RemoveContainer" containerID="cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e"
Sep 09 15:26:18.228110 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.228091    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e"} err="failed to get container status \"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e\": rpc error: code = NotFound desc = could not find container \"cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e\": container with ID starting with cd3fa18992c5028684c46753acd8d11b3953d39083253d94264e181420367b5e not found: ID does not exist"
Sep 09 15:26:18.228151 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.228111    2578 scope.go:117] "RemoveContainer" containerID="fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5"
Sep 09 15:26:18.228347 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.228331    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5"} err="failed to get container status \"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5\": rpc error: code = NotFound desc = could not find container \"fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5\": container with ID starting with fb4390bf035ae018495885946814238aa2054c35bff6f7d05f4c7f1ef4c6bbd5 not found: ID does not exist"
Sep 09 15:26:18.228347 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.228346    2578 scope.go:117] "RemoveContainer" containerID="f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16"
Sep 09 15:26:18.228591 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.228568    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16"} err="failed to get container status \"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16\": rpc error: code = NotFound desc = could not find container \"f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16\": container with ID starting with f6bb0a14506cd55c5ece5959e1073d31055714fa551a1dc0b18ce96cbd7e3e16 not found: ID does not exist"
Sep 09 15:26:18.228687 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.228670    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.231061 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.231043    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fh66r83d9f5bt\""
Sep 09 15:26:18.231144 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.231045    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 15:26:18.231640 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.231395    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 15:26:18.231640 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.231438    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6xfj9\""
Sep 09 15:26:18.231640 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.231445    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 15:26:18.231640 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.231453    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 15:26:18.231640 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.231492    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 15:26:18.231640 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.231505    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 15:26:18.231640 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.231455    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 15:26:18.231640 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.231440    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 15:26:18.231640 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.231507    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 15:26:18.231640 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.231581    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 15:26:18.232440 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.231790    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 15:26:18.232440 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.231870    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 15:26:18.232440 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.232012    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 15:26:18.234078 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.234060    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 15:26:18.237448 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.237430    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 15:26:18.242445 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.242426    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 15:26:18.344831 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.344747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-config-out\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.344831 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.344782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.344831 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.344807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.344831 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.344833    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.345150 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.344907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-config\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.345150 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.344941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.345150 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.344967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.345150 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.345031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.345150 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.345073    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.345150 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.345115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.345150 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.345129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.345150 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.345148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.345402 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.345176    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.345402 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.345196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-web-config\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.345402 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.345227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.345402 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.345269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.345402 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.345289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.345402 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.345308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.345402 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.345334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.345635 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.345405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gngjx\" (UniqueName: \"kubernetes.io/projected/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-kube-api-access-gngjx\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.446767 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.446724    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.446767 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.446767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.447017 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.446787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.447017 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.446812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.447017 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.446840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.447017 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.446948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.447017 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.447001    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.447253 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.447029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.447253 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.447060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-web-config\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.447253 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.447103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.447253 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.447128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.447253 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.447162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.447253 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.447190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.447253 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.447224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.447763 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.447285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gngjx\" (UniqueName: \"kubernetes.io/projected/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-kube-api-access-gngjx\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.447763 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.447328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-config-out\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.447763 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.447349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.447763 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.447350    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.447763 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.447370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.447763 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.447718    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.448458 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.448433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.448657 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.448636    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.448960 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.448935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.449715 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.449684    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.449972 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.449947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.450959 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.450057    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.450959 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.450234    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.450959 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.450282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.450959 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.450316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-config\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.450959 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.450456    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.451278 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.451079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.452060 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.451973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-web-config\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.452241 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.452214    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-config-out\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.452324 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.452307    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.452614 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.452595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.453068 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.453049    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.453141 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.453101    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-config\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.453597 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.453559    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.453597 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.453577    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.455258 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.455232    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gngjx\" (UniqueName: \"kubernetes.io/projected/d5b37ecf-f4b1-478c-93c9-7157e7a5a140-kube-api-access-gngjx\") pod \"prometheus-k8s-0\" (UID: \"d5b37ecf-f4b1-478c-93c9-7157e7a5a140\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.539017 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.538963    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:18.815671 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.815629    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 15:26:18.822669 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:26:18.822622    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b37ecf_f4b1_478c_93c9_7157e7a5a140.slice/crio-6b4ebd9c2a9020a09f28926d01440ff82f42038421810a808b5c5371fdcc1434 WatchSource:0}: Error finding container 6b4ebd9c2a9020a09f28926d01440ff82f42038421810a808b5c5371fdcc1434: Status 404 returned error can't find the container with id 6b4ebd9c2a9020a09f28926d01440ff82f42038421810a808b5c5371fdcc1434
Sep 09 15:26:18.865889 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:18.865856    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de3e59e7-b0e3-4238-93b6-b0531bd7e3aa" path="/var/lib/kubelet/pods/de3e59e7-b0e3-4238-93b6-b0531bd7e3aa/volumes"
Sep 09 15:26:19.173208 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:19.173178    2578 generic.go:356] "Generic (PLEG): container finished" podID="d5b37ecf-f4b1-478c-93c9-7157e7a5a140" containerID="f9741c1632678ba25fcf2940ecbe06afb395e5ed907c4602648ce6c761f5d9f3" exitCode=0
Sep 09 15:26:19.173644 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:19.173271    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5b37ecf-f4b1-478c-93c9-7157e7a5a140","Type":"ContainerDied","Data":"f9741c1632678ba25fcf2940ecbe06afb395e5ed907c4602648ce6c761f5d9f3"}
Sep 09 15:26:19.173644 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:19.173344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5b37ecf-f4b1-478c-93c9-7157e7a5a140","Type":"ContainerStarted","Data":"6b4ebd9c2a9020a09f28926d01440ff82f42038421810a808b5c5371fdcc1434"}
Sep 09 15:26:20.180955 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:20.180921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5b37ecf-f4b1-478c-93c9-7157e7a5a140","Type":"ContainerStarted","Data":"e939649fcbef77dff9a764e0e443d63a4fb886105182bb4e5f0894d1971db645"}
Sep 09 15:26:20.181399 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:20.181010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5b37ecf-f4b1-478c-93c9-7157e7a5a140","Type":"ContainerStarted","Data":"779434d0ef3992491f7881fe279c4858da88ffc70283a473e252d4eb79313456"}
Sep 09 15:26:20.181399 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:20.181043    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5b37ecf-f4b1-478c-93c9-7157e7a5a140","Type":"ContainerStarted","Data":"ae356c106227a9d55ef576d4653b53e4de9518fd3c45ef82226b29ccea9e3541"}
Sep 09 15:26:20.181399 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:20.181067    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5b37ecf-f4b1-478c-93c9-7157e7a5a140","Type":"ContainerStarted","Data":"6113e9922623ef4433e9467e97ad5ef22ec521adbf46bb178763285b2af9cc97"}
Sep 09 15:26:20.181399 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:20.181090    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5b37ecf-f4b1-478c-93c9-7157e7a5a140","Type":"ContainerStarted","Data":"d6c48dd96e842cde4ef256615e4d509acf3954bf2a63954769ddd874dcdc3ca1"}
Sep 09 15:26:20.181399 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:20.181111    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5b37ecf-f4b1-478c-93c9-7157e7a5a140","Type":"ContainerStarted","Data":"5136a675e20931bcf9e65fa9adcd934c165d8b7ff68da8357a7c3dfe02e8ff6d"}
Sep 09 15:26:20.209051 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:20.208966    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.185865551 podStartE2EDuration="2.208945131s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.07958ms" isStatefulPod=false podCreationTimestamp="2026-09-09 15:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:26:20.207268812 +0000 UTC m=+369.849310852" watchObservedRunningTime="2026-09-09 15:26:20.208945131 +0000 UTC m=+369.850987121"
Sep 09 15:26:22.190804 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:22.190777    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk" event={"ID":"1981bf38-60c0-4674-bb1e-cf6fc10eabff","Type":"ContainerStarted","Data":"d708c12a09c1bc58ca6056cc6708b68117115a99fe30db22c41ca61916d749eb"}
Sep 09 15:26:22.191133 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:22.190834    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk" event={"ID":"1981bf38-60c0-4674-bb1e-cf6fc10eabff","Type":"ContainerStarted","Data":"40ec05afab3bd95a6c5da8a107e44fc392840d6e7172d2b0fcbac949b9bc12b5"}
Sep 09 15:26:23.196828 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:23.196786    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk" event={"ID":"1981bf38-60c0-4674-bb1e-cf6fc10eabff","Type":"ContainerStarted","Data":"97be31c211d9285d8b35a1abbc1886fe7d6d90c134cf1efbd993aa8b8d8e75c1"}
Sep 09 15:26:23.217990 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:23.217932    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-797d4dcc75-8z4fk" podStartSLOduration=2.690838352 podStartE2EDuration="7.217915622s" totalImagesPullingTime="4.52707727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:26:23.216054819 +0000 UTC m=+372.858096809" watchObservedRunningTime="2026-09-09 15:26:23.217915622 +0000 UTC m=+372.859957611"
Sep 09 15:26:23.383043 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:23.383010    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:26:23.483699 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:23.483623    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:26:23.539082 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:23.539053    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:26:23.539751 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:23.539731    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:26:24.121597 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:24.121551    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:26:24.226011 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:24.225957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:26:24.606419 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:24.606378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs\") pod \"network-metrics-daemon-nhtcj\" (UID: \"7052f7a0-16d5-4115-82ae-34f07e9db9f9\") " pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:26:24.608882 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:24.608853    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7052f7a0-16d5-4115-82ae-34f07e9db9f9-metrics-certs\") pod \"network-metrics-daemon-nhtcj\" (UID: \"7052f7a0-16d5-4115-82ae-34f07e9db9f9\") " pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:26:24.764408 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:24.764376    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xdr6q\""
Sep 09 15:26:24.772273 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:24.772250    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhtcj"
Sep 09 15:26:25.040027 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:25.039963    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nhtcj"]
Sep 09 15:26:25.042721 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:26:25.042697    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7052f7a0_16d5_4115_82ae_34f07e9db9f9.slice/crio-f9769d89e612a24ff7ee9da0ff51e41e9158945bf46e203713a49f429d8f08db WatchSource:0}: Error finding container f9769d89e612a24ff7ee9da0ff51e41e9158945bf46e203713a49f429d8f08db: Status 404 returned error can't find the container with id f9769d89e612a24ff7ee9da0ff51e41e9158945bf46e203713a49f429d8f08db
Sep 09 15:26:25.207277 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:25.207241    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nhtcj" event={"ID":"7052f7a0-16d5-4115-82ae-34f07e9db9f9","Type":"ContainerStarted","Data":"f9769d89e612a24ff7ee9da0ff51e41e9158945bf46e203713a49f429d8f08db"}
Sep 09 15:26:26.216597 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:26.216523    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nhtcj" event={"ID":"7052f7a0-16d5-4115-82ae-34f07e9db9f9","Type":"ContainerStarted","Data":"1155640a6e3e37978fa72c5870e532392f79cd326f5369050e56ddc8513dca39"}
Sep 09 15:26:27.221948 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:27.221915    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nhtcj" event={"ID":"7052f7a0-16d5-4115-82ae-34f07e9db9f9","Type":"ContainerStarted","Data":"aea8ace69ba28030d2293c0774a9d76320d658c4a507afa167b3d2b5617a347c"}
Sep 09 15:26:27.237702 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:27.237655    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nhtcj" podStartSLOduration=376.187672832 podStartE2EDuration="6m17.237639433s" totalImagesPullingTime="1.049966601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:26:27.235963888 +0000 UTC m=+376.878005995" watchObservedRunningTime="2026-09-09 15:26:27.237639433 +0000 UTC m=+376.879681422"
Sep 09 15:26:29.905326 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:29.905296    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:26:30.015852 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:30.015812    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:26:31.715393 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:31.715363    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:26:31.816325 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:31.816294    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:26:34.744592 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:34.744552    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:26:34.846173 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:34.846141    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:26:34.946205 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:26:34.946172    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:27:06.217646 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:27:06.217613    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:27:18.540157 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:27:18.540122    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:27:18.556323 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:27:18.556295    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:27:19.448497 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:27:19.448471    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 15:27:27.011310 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:27:27.011268    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:27:27.146050 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:27:27.146022    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:27:32.776020 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:27:32.775923    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:27:32.828556 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:27:32.828525    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:27:38.944573 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:27:38.944534    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:27:39.047994 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:27:39.047955    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:27:44.057632 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:27:44.057594    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:27:44.157807 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:27:44.157774    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:27:44.206417 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:27:44.206387    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:27:48.824922 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:27:48.824892    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:27:48.891059 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:27:48.891032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:27:55.691178 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:27:55.691143    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:28:54.706617 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:28:54.706585    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:28:54.799391 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:28:54.799360    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:03.118361 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:03.118284    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:03.196064 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:03.196037    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:16.542856 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:16.542820    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:17.359368 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:17.359333    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:20.171171 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:20.171135    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:20.273851 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:20.273815    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:22.337391 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:22.337360    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:22.420434 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:22.420402    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:22.710973 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:22.710932    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:22.828134 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:22.828106    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:24.525765 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:24.525730    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:24.890217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:24.890184    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:27.145279 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:27.145251    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:27.245824 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:27.245801    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:31.643733 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:31.643696    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:31.743797 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:31.743771    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:32.515071 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:32.515042    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:34.684678 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:34.684639    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:34.783538 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:34.783507    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:41.414843 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:41.414807    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:42.919135 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:42.919103    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:43.096086 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:43.096059    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:55.388638 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.388608    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"]
Sep 09 15:29:55.391890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.391876    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"
Sep 09 15:29:55.394491 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.394457    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 15:29:55.394618 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.394504    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 15:29:55.394618 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.394541    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 15:29:55.395494 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.395476    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 15:29:55.395601 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.395477    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-r757k\""
Sep 09 15:29:55.395601 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.395530    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 15:29:55.400454 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.400431    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"]
Sep 09 15:29:55.484010 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.483950    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:55.531015 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.530972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5deb874a-0c47-45b8-880d-deedf83d1af2-metrics-cert\") pod \"lws-controller-manager-76f449f6db-l6jdj\" (UID: \"5deb874a-0c47-45b8-880d-deedf83d1af2\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"
Sep 09 15:29:55.531138 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.531020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5deb874a-0c47-45b8-880d-deedf83d1af2-manager-config\") pod \"lws-controller-manager-76f449f6db-l6jdj\" (UID: \"5deb874a-0c47-45b8-880d-deedf83d1af2\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"
Sep 09 15:29:55.531185 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.531160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzvdh\" (UniqueName: \"kubernetes.io/projected/5deb874a-0c47-45b8-880d-deedf83d1af2-kube-api-access-vzvdh\") pod \"lws-controller-manager-76f449f6db-l6jdj\" (UID: \"5deb874a-0c47-45b8-880d-deedf83d1af2\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"
Sep 09 15:29:55.531231 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.531188    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5deb874a-0c47-45b8-880d-deedf83d1af2-cert\") pod \"lws-controller-manager-76f449f6db-l6jdj\" (UID: \"5deb874a-0c47-45b8-880d-deedf83d1af2\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"
Sep 09 15:29:55.565107 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.565076    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:55.631755 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.631721    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzvdh\" (UniqueName: \"kubernetes.io/projected/5deb874a-0c47-45b8-880d-deedf83d1af2-kube-api-access-vzvdh\") pod \"lws-controller-manager-76f449f6db-l6jdj\" (UID: \"5deb874a-0c47-45b8-880d-deedf83d1af2\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"
Sep 09 15:29:55.631755 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.631761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5deb874a-0c47-45b8-880d-deedf83d1af2-cert\") pod \"lws-controller-manager-76f449f6db-l6jdj\" (UID: \"5deb874a-0c47-45b8-880d-deedf83d1af2\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"
Sep 09 15:29:55.631956 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.631785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5deb874a-0c47-45b8-880d-deedf83d1af2-metrics-cert\") pod \"lws-controller-manager-76f449f6db-l6jdj\" (UID: \"5deb874a-0c47-45b8-880d-deedf83d1af2\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"
Sep 09 15:29:55.631956 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.631801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5deb874a-0c47-45b8-880d-deedf83d1af2-manager-config\") pod \"lws-controller-manager-76f449f6db-l6jdj\" (UID: \"5deb874a-0c47-45b8-880d-deedf83d1af2\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"
Sep 09 15:29:55.632485 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.632459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5deb874a-0c47-45b8-880d-deedf83d1af2-manager-config\") pod \"lws-controller-manager-76f449f6db-l6jdj\" (UID: \"5deb874a-0c47-45b8-880d-deedf83d1af2\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"
Sep 09 15:29:55.634516 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.634494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5deb874a-0c47-45b8-880d-deedf83d1af2-cert\") pod \"lws-controller-manager-76f449f6db-l6jdj\" (UID: \"5deb874a-0c47-45b8-880d-deedf83d1af2\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"
Sep 09 15:29:55.634596 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.634515    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5deb874a-0c47-45b8-880d-deedf83d1af2-metrics-cert\") pod \"lws-controller-manager-76f449f6db-l6jdj\" (UID: \"5deb874a-0c47-45b8-880d-deedf83d1af2\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"
Sep 09 15:29:55.644527 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.644474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzvdh\" (UniqueName: \"kubernetes.io/projected/5deb874a-0c47-45b8-880d-deedf83d1af2-kube-api-access-vzvdh\") pod \"lws-controller-manager-76f449f6db-l6jdj\" (UID: \"5deb874a-0c47-45b8-880d-deedf83d1af2\") " pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"
Sep 09 15:29:55.702205 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.702172    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"
Sep 09 15:29:55.972475 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:55.972400    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"]
Sep 09 15:29:55.975179 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:29:55.975152    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5deb874a_0c47_45b8_880d_deedf83d1af2.slice/crio-c6ca1472a65440947c8d70fae33938a3355c9ddacd46021b4cb095f40cbb4510 WatchSource:0}: Error finding container c6ca1472a65440947c8d70fae33938a3355c9ddacd46021b4cb095f40cbb4510: Status 404 returned error can't find the container with id c6ca1472a65440947c8d70fae33938a3355c9ddacd46021b4cb095f40cbb4510
Sep 09 15:29:56.049964 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:56.049934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj" event={"ID":"5deb874a-0c47-45b8-880d-deedf83d1af2","Type":"ContainerStarted","Data":"c6ca1472a65440947c8d70fae33938a3355c9ddacd46021b4cb095f40cbb4510"}
Sep 09 15:29:59.065081 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:59.065048    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj" event={"ID":"5deb874a-0c47-45b8-880d-deedf83d1af2","Type":"ContainerStarted","Data":"6625057d37ef2604ec7cc7351c120db266c669d4f7eee3b54da1b99479423643"}
Sep 09 15:29:59.065499 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:59.065137    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"
Sep 09 15:29:59.083787 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:59.083740    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj" podStartSLOduration=1.630404529 podStartE2EDuration="4.083728027s" totalImagesPullingTime="2.453323498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:29:59.082155311 +0000 UTC m=+588.724197305" watchObservedRunningTime="2026-09-09 15:29:59.083728027 +0000 UTC m=+588.725770015"
Sep 09 15:29:59.268293 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:59.268258    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:59.564373 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:59.564333    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:29:59.814880 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:29:59.814786    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:00.018832 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.018791    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:00.166887 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.166851    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"]
Sep 09 15:30:00.170339 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.170320    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:00.174093 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.174074    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 15:30:00.174506 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.174490    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 15:30:00.174586 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.174568    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 15:30:00.174644 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.174571    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hpnwk\""
Sep 09 15:30:00.175193 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.175174    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 15:30:00.191027 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.191007    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"]
Sep 09 15:30:00.253268 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.253241    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:00.269476 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.269455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24e1e39c-65d8-4473-8696-78b2fd866d1d-tmp\") pod \"opendatahub-operator-controller-manager-74fc54996-lvfk9\" (UID: \"24e1e39c-65d8-4473-8696-78b2fd866d1d\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:00.269545 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.269485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/24e1e39c-65d8-4473-8696-78b2fd866d1d-manifests\") pod \"opendatahub-operator-controller-manager-74fc54996-lvfk9\" (UID: \"24e1e39c-65d8-4473-8696-78b2fd866d1d\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:00.269545 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.269540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/24e1e39c-65d8-4473-8696-78b2fd866d1d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74fc54996-lvfk9\" (UID: \"24e1e39c-65d8-4473-8696-78b2fd866d1d\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:00.269610 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.269563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sctc5\" (UniqueName: \"kubernetes.io/projected/24e1e39c-65d8-4473-8696-78b2fd866d1d-kube-api-access-sctc5\") pod \"opendatahub-operator-controller-manager-74fc54996-lvfk9\" (UID: \"24e1e39c-65d8-4473-8696-78b2fd866d1d\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:00.269654 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.269635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24e1e39c-65d8-4473-8696-78b2fd866d1d-webhook-cert\") pod \"opendatahub-operator-controller-manager-74fc54996-lvfk9\" (UID: \"24e1e39c-65d8-4473-8696-78b2fd866d1d\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:00.370111 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.370075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sctc5\" (UniqueName: \"kubernetes.io/projected/24e1e39c-65d8-4473-8696-78b2fd866d1d-kube-api-access-sctc5\") pod \"opendatahub-operator-controller-manager-74fc54996-lvfk9\" (UID: \"24e1e39c-65d8-4473-8696-78b2fd866d1d\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:00.370269 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.370139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24e1e39c-65d8-4473-8696-78b2fd866d1d-webhook-cert\") pod \"opendatahub-operator-controller-manager-74fc54996-lvfk9\" (UID: \"24e1e39c-65d8-4473-8696-78b2fd866d1d\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:00.370269 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.370175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24e1e39c-65d8-4473-8696-78b2fd866d1d-tmp\") pod \"opendatahub-operator-controller-manager-74fc54996-lvfk9\" (UID: \"24e1e39c-65d8-4473-8696-78b2fd866d1d\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:00.370269 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.370206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/24e1e39c-65d8-4473-8696-78b2fd866d1d-manifests\") pod \"opendatahub-operator-controller-manager-74fc54996-lvfk9\" (UID: \"24e1e39c-65d8-4473-8696-78b2fd866d1d\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:00.370385 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.370269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/24e1e39c-65d8-4473-8696-78b2fd866d1d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74fc54996-lvfk9\" (UID: \"24e1e39c-65d8-4473-8696-78b2fd866d1d\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:00.370699 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.370675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/24e1e39c-65d8-4473-8696-78b2fd866d1d-manifests\") pod \"opendatahub-operator-controller-manager-74fc54996-lvfk9\" (UID: \"24e1e39c-65d8-4473-8696-78b2fd866d1d\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:00.372717 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.372682    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24e1e39c-65d8-4473-8696-78b2fd866d1d-tmp\") pod \"opendatahub-operator-controller-manager-74fc54996-lvfk9\" (UID: \"24e1e39c-65d8-4473-8696-78b2fd866d1d\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:00.372959 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.372941    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24e1e39c-65d8-4473-8696-78b2fd866d1d-webhook-cert\") pod \"opendatahub-operator-controller-manager-74fc54996-lvfk9\" (UID: \"24e1e39c-65d8-4473-8696-78b2fd866d1d\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:00.373026 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.372973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/24e1e39c-65d8-4473-8696-78b2fd866d1d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74fc54996-lvfk9\" (UID: \"24e1e39c-65d8-4473-8696-78b2fd866d1d\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:00.382494 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.382467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sctc5\" (UniqueName: \"kubernetes.io/projected/24e1e39c-65d8-4473-8696-78b2fd866d1d-kube-api-access-sctc5\") pod \"opendatahub-operator-controller-manager-74fc54996-lvfk9\" (UID: \"24e1e39c-65d8-4473-8696-78b2fd866d1d\") " pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:00.415596 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.415566    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:00.480674 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.480589    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:00.514586 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.514554    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:00.614251 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.614215    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:00.718242 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.718209    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:00.755288 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.755264    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"]
Sep 09 15:30:00.757811 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:30:00.757787    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e1e39c_65d8_4473_8696_78b2fd866d1d.slice/crio-f059a53e0434b30e204d16dc56405eab6069e5477c77eff4b8fc801c5388c571 WatchSource:0}: Error finding container f059a53e0434b30e204d16dc56405eab6069e5477c77eff4b8fc801c5388c571: Status 404 returned error can't find the container with id f059a53e0434b30e204d16dc56405eab6069e5477c77eff4b8fc801c5388c571
Sep 09 15:30:00.820693 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.820660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:00.913723 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:00.913690    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:01.015067 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:01.014994    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:01.077326 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:01.077296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9" event={"ID":"24e1e39c-65d8-4473-8696-78b2fd866d1d","Type":"ContainerStarted","Data":"f059a53e0434b30e204d16dc56405eab6069e5477c77eff4b8fc801c5388c571"}
Sep 09 15:30:06.732692 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:06.732615    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:06.833325 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:06.833295    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:07.110368 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:07.110289    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9" event={"ID":"24e1e39c-65d8-4473-8696-78b2fd866d1d","Type":"ContainerStarted","Data":"e7abf9a5ad1f4e1b67974bd7097664dcedde610bd30b02cf2cf16e23de7a247a"}
Sep 09 15:30:08.116072 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:08.116036    2578 generic.go:356] "Generic (PLEG): container finished" podID="24e1e39c-65d8-4473-8696-78b2fd866d1d" containerID="e7abf9a5ad1f4e1b67974bd7097664dcedde610bd30b02cf2cf16e23de7a247a" exitCode=0
Sep 09 15:30:08.116466 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:08.116129    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9" event={"ID":"24e1e39c-65d8-4473-8696-78b2fd866d1d","Type":"ContainerDied","Data":"e7abf9a5ad1f4e1b67974bd7097664dcedde610bd30b02cf2cf16e23de7a247a"}
Sep 09 15:30:10.072115 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:10.072088    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-76f449f6db-l6jdj"
Sep 09 15:30:13.140101 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:13.140060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9" event={"ID":"24e1e39c-65d8-4473-8696-78b2fd866d1d","Type":"ContainerStarted","Data":"f004055b3f7dfe8225694376b145ec08dd303b591e1212ccd129ba167b445f7b"}
Sep 09 15:30:13.140511 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:13.140213    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:13.162577 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:13.162526    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9" podStartSLOduration=1.091593585 podStartE2EDuration="13.162510923s" totalImagesPullingTime="11.011439178s" totalInitContainerRuntime="1.05947816s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:30:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:30:13.160015126 +0000 UTC m=+602.802057137" watchObservedRunningTime="2026-09-09 15:30:13.162510923 +0000 UTC m=+602.804552911"
Sep 09 15:30:15.070789 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:15.070754    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 15:30:15.071447 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:15.070998    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 15:30:15.078376 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:15.078354    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:30:15.078853 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:15.078837    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:30:15.898592 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:15.898559    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:15.950970 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:15.950941    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:16.747447 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:16.747417    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:19.705374 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:19.705343    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:19.766153 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:19.766127    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:24.148034 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:24.148002    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-74fc54996-lvfk9"
Sep 09 15:30:25.726415 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:25.726385    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:25.851004 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:25.850952    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:31.384184 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.384151    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:31.479457 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.479422    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59598c6787-t5n82"]
Sep 09 15:30:31.489929 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.489900    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59598c6787-t5n82"
Sep 09 15:30:31.491106 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.491081    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59598c6787-t5n82"]
Sep 09 15:30:31.492754 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.492735    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 15:30:31.493090 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.492780    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 15:30:31.493090 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.492781    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 15:30:31.493090 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.492732    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 15:30:31.493598 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.493577    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fn29g\""
Sep 09 15:30:31.559249 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.559215    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:31.634778 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.634680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22dcf369-0bc5-4588-a4ae-c2b38cfc879f-tmp\") pod \"kube-auth-proxy-59598c6787-t5n82\" (UID: \"22dcf369-0bc5-4588-a4ae-c2b38cfc879f\") " pod="openshift-ingress/kube-auth-proxy-59598c6787-t5n82"
Sep 09 15:30:31.634778 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.634732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j77c5\" (UniqueName: \"kubernetes.io/projected/22dcf369-0bc5-4588-a4ae-c2b38cfc879f-kube-api-access-j77c5\") pod \"kube-auth-proxy-59598c6787-t5n82\" (UID: \"22dcf369-0bc5-4588-a4ae-c2b38cfc879f\") " pod="openshift-ingress/kube-auth-proxy-59598c6787-t5n82"
Sep 09 15:30:31.634941 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.634800    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22dcf369-0bc5-4588-a4ae-c2b38cfc879f-tls-certs\") pod \"kube-auth-proxy-59598c6787-t5n82\" (UID: \"22dcf369-0bc5-4588-a4ae-c2b38cfc879f\") " pod="openshift-ingress/kube-auth-proxy-59598c6787-t5n82"
Sep 09 15:30:31.735404 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.735365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22dcf369-0bc5-4588-a4ae-c2b38cfc879f-tmp\") pod \"kube-auth-proxy-59598c6787-t5n82\" (UID: \"22dcf369-0bc5-4588-a4ae-c2b38cfc879f\") " pod="openshift-ingress/kube-auth-proxy-59598c6787-t5n82"
Sep 09 15:30:31.735532 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.735449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j77c5\" (UniqueName: \"kubernetes.io/projected/22dcf369-0bc5-4588-a4ae-c2b38cfc879f-kube-api-access-j77c5\") pod \"kube-auth-proxy-59598c6787-t5n82\" (UID: \"22dcf369-0bc5-4588-a4ae-c2b38cfc879f\") " pod="openshift-ingress/kube-auth-proxy-59598c6787-t5n82"
Sep 09 15:30:31.735532 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.735513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22dcf369-0bc5-4588-a4ae-c2b38cfc879f-tls-certs\") pod \"kube-auth-proxy-59598c6787-t5n82\" (UID: \"22dcf369-0bc5-4588-a4ae-c2b38cfc879f\") " pod="openshift-ingress/kube-auth-proxy-59598c6787-t5n82"
Sep 09 15:30:31.737830 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.737802    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22dcf369-0bc5-4588-a4ae-c2b38cfc879f-tmp\") pod \"kube-auth-proxy-59598c6787-t5n82\" (UID: \"22dcf369-0bc5-4588-a4ae-c2b38cfc879f\") " pod="openshift-ingress/kube-auth-proxy-59598c6787-t5n82"
Sep 09 15:30:31.738008 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.737971    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22dcf369-0bc5-4588-a4ae-c2b38cfc879f-tls-certs\") pod \"kube-auth-proxy-59598c6787-t5n82\" (UID: \"22dcf369-0bc5-4588-a4ae-c2b38cfc879f\") " pod="openshift-ingress/kube-auth-proxy-59598c6787-t5n82"
Sep 09 15:30:31.744382 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.744360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j77c5\" (UniqueName: \"kubernetes.io/projected/22dcf369-0bc5-4588-a4ae-c2b38cfc879f-kube-api-access-j77c5\") pod \"kube-auth-proxy-59598c6787-t5n82\" (UID: \"22dcf369-0bc5-4588-a4ae-c2b38cfc879f\") " pod="openshift-ingress/kube-auth-proxy-59598c6787-t5n82"
Sep 09 15:30:31.802755 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:31.802727    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59598c6787-t5n82"
Sep 09 15:30:32.076056 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:32.076028    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59598c6787-t5n82"]
Sep 09 15:30:32.078686 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:30:32.078658    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22dcf369_0bc5_4588_a4ae_c2b38cfc879f.slice/crio-107eb9895632ad48c5765dfad8d45ee5b6fbc36244083b0ee8e46b7ea4256199 WatchSource:0}: Error finding container 107eb9895632ad48c5765dfad8d45ee5b6fbc36244083b0ee8e46b7ea4256199: Status 404 returned error can't find the container with id 107eb9895632ad48c5765dfad8d45ee5b6fbc36244083b0ee8e46b7ea4256199
Sep 09 15:30:32.224107 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:32.224073    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59598c6787-t5n82" event={"ID":"22dcf369-0bc5-4588-a4ae-c2b38cfc879f","Type":"ContainerStarted","Data":"107eb9895632ad48c5765dfad8d45ee5b6fbc36244083b0ee8e46b7ea4256199"}
Sep 09 15:30:32.485741 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:32.485711    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:32.557375 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:32.557348    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:33.386327 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:33.386291    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:35.242288 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:35.242253    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59598c6787-t5n82" event={"ID":"22dcf369-0bc5-4588-a4ae-c2b38cfc879f","Type":"ContainerStarted","Data":"0a01535f1824a949e3ec9f8af430caa26f79f2624a34c6ad63a4cb5a673405dd"}
Sep 09 15:30:35.258481 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:35.258435    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59598c6787-t5n82" podStartSLOduration=1.296793898 podStartE2EDuration="4.258420859s" totalImagesPullingTime="2.961626961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:30:35.256857595 +0000 UTC m=+624.898899583" watchObservedRunningTime="2026-09-09 15:30:35.258420859 +0000 UTC m=+624.900462888"
Sep 09 15:30:37.047560 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:37.047530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:37.146648 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:37.146615    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:37.248320 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:37.248283    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:37.431823 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:37.431778    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:37.595839 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:37.595786    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:37.689197 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:37.689117    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:37.768947 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:37.768914    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:38.387258 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:38.387229    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:38.634993 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:38.634959    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:38.734394 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:38.734320    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:38.935778 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:38.935745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:39.033004 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:39.032874    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:46.764532 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:46.764478    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:46.886779 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:46.886732    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:47.096336 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:47.096193    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:47.410821 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:47.410769    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:48.070344 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:48.070306    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:48.171538 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:48.171495    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:30:48.289828 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:30:48.289751    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:31:25.144098 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:31:25.144061    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:31:37.796783 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:31:37.796724    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:31:37.894906 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:31:37.894865    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:31:45.412345 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:31:45.412291    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:31:48.579391 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:31:48.579353    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:31:48.680736 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:31:48.680711    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:31:52.543210 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:31:52.543176    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:31:52.643606 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:31:52.643573    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:31:56.434648 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:31:56.434620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:31:58.866658 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:31:58.866627    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:31:59.313772 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:31:59.313739    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:01.101137 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:01.101100    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:01.204839 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:01.204801    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:01.898215 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:01.898122    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:02.462423 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:02.462375    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:03.400923 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:03.400888    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-snwd7"]
Sep 09 15:32:03.403430 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:03.403412    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-snwd7"
Sep 09 15:32:03.406412 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:03.406354    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 15:32:03.406412 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:03.406391    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 09 15:32:03.406630 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:03.406445    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-97sq8\""
Sep 09 15:32:03.407179 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:03.407163    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 15:32:03.422646 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:03.422621    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-snwd7"]
Sep 09 15:32:03.523695 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:03.523653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nszgs\" (UniqueName: \"kubernetes.io/projected/88fb0e01-a115-4c3b-b495-fe57bddfd9e0-kube-api-access-nszgs\") pod \"dns-operator-controller-manager-7b84cd7c9c-snwd7\" (UID: \"88fb0e01-a115-4c3b-b495-fe57bddfd9e0\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-snwd7"
Sep 09 15:32:03.624437 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:03.624404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nszgs\" (UniqueName: \"kubernetes.io/projected/88fb0e01-a115-4c3b-b495-fe57bddfd9e0-kube-api-access-nszgs\") pod \"dns-operator-controller-manager-7b84cd7c9c-snwd7\" (UID: \"88fb0e01-a115-4c3b-b495-fe57bddfd9e0\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-snwd7"
Sep 09 15:32:03.647668 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:03.647633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nszgs\" (UniqueName: \"kubernetes.io/projected/88fb0e01-a115-4c3b-b495-fe57bddfd9e0-kube-api-access-nszgs\") pod \"dns-operator-controller-manager-7b84cd7c9c-snwd7\" (UID: \"88fb0e01-a115-4c3b-b495-fe57bddfd9e0\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-snwd7"
Sep 09 15:32:03.713600 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:03.713499    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-snwd7"
Sep 09 15:32:03.728890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:03.728861    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:03.833122 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:03.833092    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:03.993423 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:03.993339    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-snwd7"]
Sep 09 15:32:03.996556 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:32:03.996527    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88fb0e01_a115_4c3b_b495_fe57bddfd9e0.slice/crio-72dee847b353ed14aa843547c063cfd078543886ffb3aa4f92d266f2314925d5 WatchSource:0}: Error finding container 72dee847b353ed14aa843547c063cfd078543886ffb3aa4f92d266f2314925d5: Status 404 returned error can't find the container with id 72dee847b353ed14aa843547c063cfd078543886ffb3aa4f92d266f2314925d5
Sep 09 15:32:03.998656 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:03.998640    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 15:32:04.433859 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:04.433818    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:04.639246 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:04.639217    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-snwd7" event={"ID":"88fb0e01-a115-4c3b-b495-fe57bddfd9e0","Type":"ContainerStarted","Data":"72dee847b353ed14aa843547c063cfd078543886ffb3aa4f92d266f2314925d5"}
Sep 09 15:32:04.684869 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:04.684748    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:04.785103 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:04.785070    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:04.885117 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:04.885078    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:06.652146 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:06.652108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-snwd7" event={"ID":"88fb0e01-a115-4c3b-b495-fe57bddfd9e0","Type":"ContainerStarted","Data":"b873a6e4fd3852d5f9406594048fbf04d7cc855cf55aa96a177e98c5753988cf"}
Sep 09 15:32:06.652549 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:06.652237    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-snwd7"
Sep 09 15:32:06.671810 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:06.671760    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-snwd7" podStartSLOduration=1.116973431 podStartE2EDuration="3.671745834s" totalImagesPullingTime="2.554772403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:32:06.670233886 +0000 UTC m=+716.312275896" watchObservedRunningTime="2026-09-09 15:32:06.671745834 +0000 UTC m=+716.313787822"
Sep 09 15:32:07.552545 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:07.552511    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"]
Sep 09 15:32:07.555873 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:07.555857    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"
Sep 09 15:32:07.558939 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:07.558920    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4zwbk\""
Sep 09 15:32:07.571943 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:07.571916    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"]
Sep 09 15:32:07.663331 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:07.663297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw8wm\" (UniqueName: \"kubernetes.io/projected/2d41c24d-336f-4dc3-a802-313e2925a224-kube-api-access-rw8wm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-trjrx\" (UID: \"2d41c24d-336f-4dc3-a802-313e2925a224\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"
Sep 09 15:32:07.663696 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:07.663336    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d41c24d-336f-4dc3-a802-313e2925a224-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-trjrx\" (UID: \"2d41c24d-336f-4dc3-a802-313e2925a224\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"
Sep 09 15:32:07.764000 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:07.763941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw8wm\" (UniqueName: \"kubernetes.io/projected/2d41c24d-336f-4dc3-a802-313e2925a224-kube-api-access-rw8wm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-trjrx\" (UID: \"2d41c24d-336f-4dc3-a802-313e2925a224\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"
Sep 09 15:32:07.764199 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:07.764025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d41c24d-336f-4dc3-a802-313e2925a224-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-trjrx\" (UID: \"2d41c24d-336f-4dc3-a802-313e2925a224\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"
Sep 09 15:32:07.764443 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:07.764418    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d41c24d-336f-4dc3-a802-313e2925a224-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-trjrx\" (UID: \"2d41c24d-336f-4dc3-a802-313e2925a224\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"
Sep 09 15:32:07.772115 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:07.772097    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw8wm\" (UniqueName: \"kubernetes.io/projected/2d41c24d-336f-4dc3-a802-313e2925a224-kube-api-access-rw8wm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-trjrx\" (UID: \"2d41c24d-336f-4dc3-a802-313e2925a224\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"
Sep 09 15:32:07.865530 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:07.865444    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"
Sep 09 15:32:08.139398 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:08.139370    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"]
Sep 09 15:32:08.141096 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:32:08.141070    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d41c24d_336f_4dc3_a802_313e2925a224.slice/crio-72c61efac919588b0ad215424a0433dbf4d7aee116fd5a6291ea46373c086285 WatchSource:0}: Error finding container 72c61efac919588b0ad215424a0433dbf4d7aee116fd5a6291ea46373c086285: Status 404 returned error can't find the container with id 72c61efac919588b0ad215424a0433dbf4d7aee116fd5a6291ea46373c086285
Sep 09 15:32:08.662243 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:08.662209    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx" event={"ID":"2d41c24d-336f-4dc3-a802-313e2925a224","Type":"ContainerStarted","Data":"72c61efac919588b0ad215424a0433dbf4d7aee116fd5a6291ea46373c086285"}
Sep 09 15:32:09.865255 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:09.865225    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:09.970031 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:09.970002    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:13.453190 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:13.453153    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:13.553109 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:13.553065    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:15.927873 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:15.927838    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:15.990205 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:15.990171    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:16.709961 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:16.709925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx" event={"ID":"2d41c24d-336f-4dc3-a802-313e2925a224","Type":"ContainerStarted","Data":"2281168c5a40eeaf2e96bec58b258bcfac958df401807e9c8fb660c4cdbaa006"}
Sep 09 15:32:16.710183 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:16.710033    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"
Sep 09 15:32:16.734004 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:16.733934    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx" podStartSLOduration=1.511235796 podStartE2EDuration="9.733920483s" totalImagesPullingTime="8.222684687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:32:16.732110279 +0000 UTC m=+726.374152297" watchObservedRunningTime="2026-09-09 15:32:16.733920483 +0000 UTC m=+726.375962481"
Sep 09 15:32:17.659725 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:17.659697    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-snwd7"
Sep 09 15:32:19.040022 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:19.039956    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:19.345321 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:19.345231    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:22.397601 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:22.397571    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:25.670311 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:25.670281    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:27.718080 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:27.718053    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"
Sep 09 15:32:29.047123 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:29.047096    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:29.150754 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:29.150720    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:29.798002 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:29.797955    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"]
Sep 09 15:32:29.798168 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:29.798140    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx" podUID="2d41c24d-336f-4dc3-a802-313e2925a224" containerName="manager" containerID="cri-o://2281168c5a40eeaf2e96bec58b258bcfac958df401807e9c8fb660c4cdbaa006" gracePeriod=2
Sep 09 15:32:29.808506 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:29.808477    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"]
Sep 09 15:32:29.818675 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:32:29.818632    2578 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"2d41c24d-336f-4dc3-a802-313e2925a224\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-trjrx\": pods \"kuadrant-operator-controller-manager-76f8f686bf-trjrx\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"
Sep 09 15:32:29.822547 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:29.822521    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp"]
Sep 09 15:32:29.823168 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:29.823151    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d41c24d-336f-4dc3-a802-313e2925a224" containerName="manager"
Sep 09 15:32:29.823218 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:29.823173    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d41c24d-336f-4dc3-a802-313e2925a224" containerName="manager"
Sep 09 15:32:29.823272 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:29.823262    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d41c24d-336f-4dc3-a802-313e2925a224" containerName="manager"
Sep 09 15:32:29.826509 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:29.826492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp"
Sep 09 15:32:29.846647 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:29.846621    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp"]
Sep 09 15:32:29.855290 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:29.855266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5ad5beb-73a9-41fd-9bc4-10946219de94-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-z25vp\" (UID: \"e5ad5beb-73a9-41fd-9bc4-10946219de94\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp"
Sep 09 15:32:29.855434 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:29.855295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dppjl\" (UniqueName: \"kubernetes.io/projected/e5ad5beb-73a9-41fd-9bc4-10946219de94-kube-api-access-dppjl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-z25vp\" (UID: \"e5ad5beb-73a9-41fd-9bc4-10946219de94\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp"
Sep 09 15:32:29.861930 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:32:29.861903    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-trjrx\" is forbidden: User \"system:node:ip-10-0-134-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-183.ec2.internal' and this object" podUID="2d41c24d-336f-4dc3-a802-313e2925a224" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"
Sep 09 15:32:29.956376 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:29.956336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5ad5beb-73a9-41fd-9bc4-10946219de94-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-z25vp\" (UID: \"e5ad5beb-73a9-41fd-9bc4-10946219de94\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp"
Sep 09 15:32:29.956590 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:29.956390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dppjl\" (UniqueName: \"kubernetes.io/projected/e5ad5beb-73a9-41fd-9bc4-10946219de94-kube-api-access-dppjl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-z25vp\" (UID: \"e5ad5beb-73a9-41fd-9bc4-10946219de94\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp"
Sep 09 15:32:29.957100 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:29.957068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5ad5beb-73a9-41fd-9bc4-10946219de94-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-z25vp\" (UID: \"e5ad5beb-73a9-41fd-9bc4-10946219de94\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp"
Sep 09 15:32:29.966611 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:29.966587    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dppjl\" (UniqueName: \"kubernetes.io/projected/e5ad5beb-73a9-41fd-9bc4-10946219de94-kube-api-access-dppjl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-z25vp\" (UID: \"e5ad5beb-73a9-41fd-9bc4-10946219de94\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp"
Sep 09 15:32:30.035229 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.035206    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"
Sep 09 15:32:30.037446 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:32:30.037419    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-trjrx\" is forbidden: User \"system:node:ip-10-0-134-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-183.ec2.internal' and this object" podUID="2d41c24d-336f-4dc3-a802-313e2925a224" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"
Sep 09 15:32:30.058406 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.058350    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d41c24d-336f-4dc3-a802-313e2925a224-extensions-socket-volume\") pod \"2d41c24d-336f-4dc3-a802-313e2925a224\" (UID: \"2d41c24d-336f-4dc3-a802-313e2925a224\") "
Sep 09 15:32:30.058762 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.058441    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw8wm\" (UniqueName: \"kubernetes.io/projected/2d41c24d-336f-4dc3-a802-313e2925a224-kube-api-access-rw8wm\") pod \"2d41c24d-336f-4dc3-a802-313e2925a224\" (UID: \"2d41c24d-336f-4dc3-a802-313e2925a224\") "
Sep 09 15:32:30.058922 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.058886    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d41c24d-336f-4dc3-a802-313e2925a224-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2d41c24d-336f-4dc3-a802-313e2925a224" (UID: "2d41c24d-336f-4dc3-a802-313e2925a224"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:32:30.060661 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.060639    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d41c24d-336f-4dc3-a802-313e2925a224-kube-api-access-rw8wm" (OuterVolumeSpecName: "kube-api-access-rw8wm") pod "2d41c24d-336f-4dc3-a802-313e2925a224" (UID: "2d41c24d-336f-4dc3-a802-313e2925a224"). InnerVolumeSpecName "kube-api-access-rw8wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:32:30.159523 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.159490    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw8wm\" (UniqueName: \"kubernetes.io/projected/2d41c24d-336f-4dc3-a802-313e2925a224-kube-api-access-rw8wm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:32:30.159523 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.159517    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d41c24d-336f-4dc3-a802-313e2925a224-extensions-socket-volume\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:32:30.175432 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.175402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp"
Sep 09 15:32:30.452896 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.452873    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp"]
Sep 09 15:32:30.454926 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:32:30.454890    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ad5beb_73a9_41fd_9bc4_10946219de94.slice/crio-fe0b315c91578c850c79555b575e88faa9a4638b4edda56369ae83959b856c27 WatchSource:0}: Error finding container fe0b315c91578c850c79555b575e88faa9a4638b4edda56369ae83959b856c27: Status 404 returned error can't find the container with id fe0b315c91578c850c79555b575e88faa9a4638b4edda56369ae83959b856c27
Sep 09 15:32:30.776662 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.776517    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp" event={"ID":"e5ad5beb-73a9-41fd-9bc4-10946219de94","Type":"ContainerStarted","Data":"112ec0b6e04d39934b6e0317380cdcb4de0ede3d12b295eaaf463c1e42a1bd6d"}
Sep 09 15:32:30.776662 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.776589    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp"
Sep 09 15:32:30.776662 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.776615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp" event={"ID":"e5ad5beb-73a9-41fd-9bc4-10946219de94","Type":"ContainerStarted","Data":"fe0b315c91578c850c79555b575e88faa9a4638b4edda56369ae83959b856c27"}
Sep 09 15:32:30.777960 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.777933    2578 generic.go:356] "Generic (PLEG): container finished" podID="2d41c24d-336f-4dc3-a802-313e2925a224" containerID="2281168c5a40eeaf2e96bec58b258bcfac958df401807e9c8fb660c4cdbaa006" exitCode=0
Sep 09 15:32:30.778101 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.777972    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"
Sep 09 15:32:30.778101 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.778031    2578 scope.go:117] "RemoveContainer" containerID="2281168c5a40eeaf2e96bec58b258bcfac958df401807e9c8fb660c4cdbaa006"
Sep 09 15:32:30.778775 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:32:30.778744    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-trjrx\" is forbidden: User \"system:node:ip-10-0-134-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-183.ec2.internal' and this object" podUID="2d41c24d-336f-4dc3-a802-313e2925a224" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"
Sep 09 15:32:30.787127 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.787107    2578 scope.go:117] "RemoveContainer" containerID="2281168c5a40eeaf2e96bec58b258bcfac958df401807e9c8fb660c4cdbaa006"
Sep 09 15:32:30.787405 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:32:30.787388    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2281168c5a40eeaf2e96bec58b258bcfac958df401807e9c8fb660c4cdbaa006\": container with ID starting with 2281168c5a40eeaf2e96bec58b258bcfac958df401807e9c8fb660c4cdbaa006 not found: ID does not exist" containerID="2281168c5a40eeaf2e96bec58b258bcfac958df401807e9c8fb660c4cdbaa006"
Sep 09 15:32:30.787481 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.787413    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2281168c5a40eeaf2e96bec58b258bcfac958df401807e9c8fb660c4cdbaa006"} err="failed to get container status \"2281168c5a40eeaf2e96bec58b258bcfac958df401807e9c8fb660c4cdbaa006\": rpc error: code = NotFound desc = could not find container \"2281168c5a40eeaf2e96bec58b258bcfac958df401807e9c8fb660c4cdbaa006\": container with ID starting with 2281168c5a40eeaf2e96bec58b258bcfac958df401807e9c8fb660c4cdbaa006 not found: ID does not exist"
Sep 09 15:32:30.805705 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.805657    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp" podStartSLOduration=1.80561698 podStartE2EDuration="1.80561698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:32:30.80268316 +0000 UTC m=+740.444725160" watchObservedRunningTime="2026-09-09 15:32:30.80561698 +0000 UTC m=+740.447658968"
Sep 09 15:32:30.805850 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:32:30.805806    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-trjrx\" is forbidden: User \"system:node:ip-10-0-134-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-183.ec2.internal' and this object" podUID="2d41c24d-336f-4dc3-a802-313e2925a224" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"
Sep 09 15:32:30.865043 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:32:30.865012    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-trjrx\" is forbidden: User \"system:node:ip-10-0-134-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-183.ec2.internal' and this object" podUID="2d41c24d-336f-4dc3-a802-313e2925a224" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-trjrx"
Sep 09 15:32:30.866622 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:30.866600    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d41c24d-336f-4dc3-a802-313e2925a224" path="/var/lib/kubelet/pods/2d41c24d-336f-4dc3-a802-313e2925a224/volumes"
Sep 09 15:32:32.639158 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:32.639129    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:32.736073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:32.736025    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:32.836374 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:32.836345    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:32.889444 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:32.889358    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:35.613698 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:35.613654    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:35.662328 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:35.662298    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:41.786304 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:41.786270    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp"
Sep 09 15:32:41.972762 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:41.972717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:42.073218 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:42.073119    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:42.146370 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:42.146339    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:45.478081 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:45.478046    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp"]
Sep 09 15:32:45.478461 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:45.478259    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp" podUID="e5ad5beb-73a9-41fd-9bc4-10946219de94" containerName="manager" containerID="cri-o://112ec0b6e04d39934b6e0317380cdcb4de0ede3d12b295eaaf463c1e42a1bd6d" gracePeriod=10
Sep 09 15:32:45.719047 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:45.719017    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp"
Sep 09 15:32:45.792772 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:45.792683    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5ad5beb-73a9-41fd-9bc4-10946219de94-extensions-socket-volume\") pod \"e5ad5beb-73a9-41fd-9bc4-10946219de94\" (UID: \"e5ad5beb-73a9-41fd-9bc4-10946219de94\") "
Sep 09 15:32:45.792772 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:45.792733    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dppjl\" (UniqueName: \"kubernetes.io/projected/e5ad5beb-73a9-41fd-9bc4-10946219de94-kube-api-access-dppjl\") pod \"e5ad5beb-73a9-41fd-9bc4-10946219de94\" (UID: \"e5ad5beb-73a9-41fd-9bc4-10946219de94\") "
Sep 09 15:32:45.793186 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:45.793159    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ad5beb-73a9-41fd-9bc4-10946219de94-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e5ad5beb-73a9-41fd-9bc4-10946219de94" (UID: "e5ad5beb-73a9-41fd-9bc4-10946219de94"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:32:45.795011 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:45.794966    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ad5beb-73a9-41fd-9bc4-10946219de94-kube-api-access-dppjl" (OuterVolumeSpecName: "kube-api-access-dppjl") pod "e5ad5beb-73a9-41fd-9bc4-10946219de94" (UID: "e5ad5beb-73a9-41fd-9bc4-10946219de94"). InnerVolumeSpecName "kube-api-access-dppjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:32:45.850147 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:45.850117    2578 generic.go:356] "Generic (PLEG): container finished" podID="e5ad5beb-73a9-41fd-9bc4-10946219de94" containerID="112ec0b6e04d39934b6e0317380cdcb4de0ede3d12b295eaaf463c1e42a1bd6d" exitCode=0
Sep 09 15:32:45.850328 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:45.850180    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp"
Sep 09 15:32:45.850328 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:45.850210    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp" event={"ID":"e5ad5beb-73a9-41fd-9bc4-10946219de94","Type":"ContainerDied","Data":"112ec0b6e04d39934b6e0317380cdcb4de0ede3d12b295eaaf463c1e42a1bd6d"}
Sep 09 15:32:45.850328 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:45.850267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp" event={"ID":"e5ad5beb-73a9-41fd-9bc4-10946219de94","Type":"ContainerDied","Data":"fe0b315c91578c850c79555b575e88faa9a4638b4edda56369ae83959b856c27"}
Sep 09 15:32:45.850328 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:45.850290    2578 scope.go:117] "RemoveContainer" containerID="112ec0b6e04d39934b6e0317380cdcb4de0ede3d12b295eaaf463c1e42a1bd6d"
Sep 09 15:32:45.861450 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:45.861345    2578 scope.go:117] "RemoveContainer" containerID="112ec0b6e04d39934b6e0317380cdcb4de0ede3d12b295eaaf463c1e42a1bd6d"
Sep 09 15:32:45.862030 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:32:45.862009    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"112ec0b6e04d39934b6e0317380cdcb4de0ede3d12b295eaaf463c1e42a1bd6d\": container with ID starting with 112ec0b6e04d39934b6e0317380cdcb4de0ede3d12b295eaaf463c1e42a1bd6d not found: ID does not exist" containerID="112ec0b6e04d39934b6e0317380cdcb4de0ede3d12b295eaaf463c1e42a1bd6d"
Sep 09 15:32:45.862111 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:45.862041    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"112ec0b6e04d39934b6e0317380cdcb4de0ede3d12b295eaaf463c1e42a1bd6d"} err="failed to get container status \"112ec0b6e04d39934b6e0317380cdcb4de0ede3d12b295eaaf463c1e42a1bd6d\": rpc error: code = NotFound desc = could not find container \"112ec0b6e04d39934b6e0317380cdcb4de0ede3d12b295eaaf463c1e42a1bd6d\": container with ID starting with 112ec0b6e04d39934b6e0317380cdcb4de0ede3d12b295eaaf463c1e42a1bd6d not found: ID does not exist"
Sep 09 15:32:45.871842 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:45.871819    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp"]
Sep 09 15:32:45.878641 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:45.878622    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z25vp"]
Sep 09 15:32:45.893429 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:45.893412    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dppjl\" (UniqueName: \"kubernetes.io/projected/e5ad5beb-73a9-41fd-9bc4-10946219de94-kube-api-access-dppjl\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:32:45.893516 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:45.893431    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5ad5beb-73a9-41fd-9bc4-10946219de94-extensions-socket-volume\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:32:46.866602 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:46.866574    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5ad5beb-73a9-41fd-9bc4-10946219de94" path="/var/lib/kubelet/pods/e5ad5beb-73a9-41fd-9bc4-10946219de94/volumes"
Sep 09 15:32:56.537934 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:56.537895    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:32:56.600693 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:32:56.600659    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:33:01.827734 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:01.827698    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"]
Sep 09 15:33:01.828126 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:01.828110    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5ad5beb-73a9-41fd-9bc4-10946219de94" containerName="manager"
Sep 09 15:33:01.828175 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:01.828130    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5ad5beb-73a9-41fd-9bc4-10946219de94" containerName="manager"
Sep 09 15:33:01.828213 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:01.828187    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5ad5beb-73a9-41fd-9bc4-10946219de94" containerName="manager"
Sep 09 15:33:01.830706 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:01.830690    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:01.833313 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:01.833280    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-jzqzm\""
Sep 09 15:33:01.833429 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:01.833320    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 15:33:01.842857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:01.842832    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"]
Sep 09 15:33:01.863512 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:01.863462    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:33:01.932168 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:01.932139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41c51d4-c973-4e22-a0a1-96c523e020f3-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:01.932168 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:01.932170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41c51d4-c973-4e22-a0a1-96c523e020f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:01.932357 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:01.932204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41c51d4-c973-4e22-a0a1-96c523e020f3-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:01.932357 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:01.932219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41c51d4-c973-4e22-a0a1-96c523e020f3-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:01.932357 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:01.932300    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41c51d4-c973-4e22-a0a1-96c523e020f3-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:01.932458 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:01.932403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41c51d4-c973-4e22-a0a1-96c523e020f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:01.932458 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:01.932425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41c51d4-c973-4e22-a0a1-96c523e020f3-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:01.932458 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:01.932450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41c51d4-c973-4e22-a0a1-96c523e020f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:01.932557 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:01.932472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt7nl\" (UniqueName: \"kubernetes.io/projected/d41c51d4-c973-4e22-a0a1-96c523e020f3-kube-api-access-tt7nl\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.033778 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.033745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41c51d4-c973-4e22-a0a1-96c523e020f3-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.033778 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.033781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41c51d4-c973-4e22-a0a1-96c523e020f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.034015 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.033825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41c51d4-c973-4e22-a0a1-96c523e020f3-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.034015 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.033847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41c51d4-c973-4e22-a0a1-96c523e020f3-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.034015 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.033889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41c51d4-c973-4e22-a0a1-96c523e020f3-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.034015 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.033916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41c51d4-c973-4e22-a0a1-96c523e020f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.034015 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.033939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41c51d4-c973-4e22-a0a1-96c523e020f3-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.034015 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.033964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41c51d4-c973-4e22-a0a1-96c523e020f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.034015 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.034015    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt7nl\" (UniqueName: \"kubernetes.io/projected/d41c51d4-c973-4e22-a0a1-96c523e020f3-kube-api-access-tt7nl\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.034357 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.034321    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d41c51d4-c973-4e22-a0a1-96c523e020f3-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.034419 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.034387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41c51d4-c973-4e22-a0a1-96c523e020f3-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.034419 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.034401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d41c51d4-c973-4e22-a0a1-96c523e020f3-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.034615 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.034595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d41c51d4-c973-4e22-a0a1-96c523e020f3-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.034686 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.034615    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d41c51d4-c973-4e22-a0a1-96c523e020f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.036527 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.036504    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d41c51d4-c973-4e22-a0a1-96c523e020f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.036764 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.036744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d41c51d4-c973-4e22-a0a1-96c523e020f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.047965 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.047937    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d41c51d4-c973-4e22-a0a1-96c523e020f3-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.051002 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.050955    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt7nl\" (UniqueName: \"kubernetes.io/projected/d41c51d4-c973-4e22-a0a1-96c523e020f3-kube-api-access-tt7nl\") pod \"maas-default-gateway-openshift-default-78bb78ff78-c784m\" (UID: \"d41c51d4-c973-4e22-a0a1-96c523e020f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.073763 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.073733    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:33:02.141120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.141091    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:02.413571 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.413548    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"]
Sep 09 15:33:02.415687 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:33:02.415661    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41c51d4_c973_4e22_a0a1_96c523e020f3.slice/crio-6d431935a1e3e358527e36cf5c0e8aae9f46084f12ce4284a20a5d2df8dcf841 WatchSource:0}: Error finding container 6d431935a1e3e358527e36cf5c0e8aae9f46084f12ce4284a20a5d2df8dcf841: Status 404 returned error can't find the container with id 6d431935a1e3e358527e36cf5c0e8aae9f46084f12ce4284a20a5d2df8dcf841
Sep 09 15:33:02.933608 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:02.933569    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m" event={"ID":"d41c51d4-c973-4e22-a0a1-96c523e020f3","Type":"ContainerStarted","Data":"6d431935a1e3e358527e36cf5c0e8aae9f46084f12ce4284a20a5d2df8dcf841"}
Sep 09 15:33:05.231119 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:05.231076    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 15:33:05.231413 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:05.231157    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 15:33:05.231413 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:05.231189    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 15:33:05.950189 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:05.950154    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m" event={"ID":"d41c51d4-c973-4e22-a0a1-96c523e020f3","Type":"ContainerStarted","Data":"93615f450ef1a07089dca273f978e0c139bb6a43c2009ff7cedb851f129f9faf"}
Sep 09 15:33:05.968714 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:05.968669    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m" podStartSLOduration=2.1553938759999998 podStartE2EDuration="4.968652886s" totalImagesPullingTime="2.81325901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:33:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:33:05.967533357 +0000 UTC m=+775.609575367" watchObservedRunningTime="2026-09-09 15:33:05.968652886 +0000 UTC m=+775.610694943"
Sep 09 15:33:06.141920 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:06.141883    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:06.147219 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:06.147190    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:06.954876 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:06.954847    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:06.955844 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:06.955822    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:33:12.110829 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:12.110795    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:33:21.167219 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.167185    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:33:21.177399 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.177370    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8rbfz"]
Sep 09 15:33:21.182091 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.182075    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-8rbfz"
Sep 09 15:33:21.184348 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.184326    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 15:33:21.184459 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.184384    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6768q\""
Sep 09 15:33:21.187635 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.187614    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8rbfz"]
Sep 09 15:33:21.200271 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.200247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv8m6\" (UniqueName: \"kubernetes.io/projected/4e149c93-225f-4e52-95f8-3377915cb90d-kube-api-access-dv8m6\") pod \"limitador-limitador-8588749977-8rbfz\" (UID: \"4e149c93-225f-4e52-95f8-3377915cb90d\") " pod="kuadrant-system/limitador-limitador-8588749977-8rbfz"
Sep 09 15:33:21.200364 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.200284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4e149c93-225f-4e52-95f8-3377915cb90d-config-file\") pod \"limitador-limitador-8588749977-8rbfz\" (UID: \"4e149c93-225f-4e52-95f8-3377915cb90d\") " pod="kuadrant-system/limitador-limitador-8588749977-8rbfz"
Sep 09 15:33:21.273034 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.273006    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8rbfz"]
Sep 09 15:33:21.278716 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.278694    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:33:21.300815 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.300792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv8m6\" (UniqueName: \"kubernetes.io/projected/4e149c93-225f-4e52-95f8-3377915cb90d-kube-api-access-dv8m6\") pod \"limitador-limitador-8588749977-8rbfz\" (UID: \"4e149c93-225f-4e52-95f8-3377915cb90d\") " pod="kuadrant-system/limitador-limitador-8588749977-8rbfz"
Sep 09 15:33:21.300960 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.300839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4e149c93-225f-4e52-95f8-3377915cb90d-config-file\") pod \"limitador-limitador-8588749977-8rbfz\" (UID: \"4e149c93-225f-4e52-95f8-3377915cb90d\") " pod="kuadrant-system/limitador-limitador-8588749977-8rbfz"
Sep 09 15:33:21.301486 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.301469    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4e149c93-225f-4e52-95f8-3377915cb90d-config-file\") pod \"limitador-limitador-8588749977-8rbfz\" (UID: \"4e149c93-225f-4e52-95f8-3377915cb90d\") " pod="kuadrant-system/limitador-limitador-8588749977-8rbfz"
Sep 09 15:33:21.308458 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.308430    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv8m6\" (UniqueName: \"kubernetes.io/projected/4e149c93-225f-4e52-95f8-3377915cb90d-kube-api-access-dv8m6\") pod \"limitador-limitador-8588749977-8rbfz\" (UID: \"4e149c93-225f-4e52-95f8-3377915cb90d\") " pod="kuadrant-system/limitador-limitador-8588749977-8rbfz"
Sep 09 15:33:21.337576 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.337553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:33:21.492916 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.492826    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-8rbfz"
Sep 09 15:33:21.762440 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.762416    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8rbfz"]
Sep 09 15:33:21.764621 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:33:21.764591    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e149c93_225f_4e52_95f8_3377915cb90d.slice/crio-ef7ab116ff5cfdb644daf9e3592d900312f523cff67f1e648c46e3bcda50fc92 WatchSource:0}: Error finding container ef7ab116ff5cfdb644daf9e3592d900312f523cff67f1e648c46e3bcda50fc92: Status 404 returned error can't find the container with id ef7ab116ff5cfdb644daf9e3592d900312f523cff67f1e648c46e3bcda50fc92
Sep 09 15:33:21.960663 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.960634    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-44hmv"]
Sep 09 15:33:21.965621 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.965605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-44hmv"
Sep 09 15:33:21.968965 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.968942    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9frgw\""
Sep 09 15:33:21.970627 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.970605    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-44hmv"]
Sep 09 15:33:21.991954 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:21.991926    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:33:22.006206 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:22.006187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx9dc\" (UniqueName: \"kubernetes.io/projected/eac94ec2-102e-481e-a744-c17f25a49374-kube-api-access-rx9dc\") pod \"authorino-7bcfd8f9cd-44hmv\" (UID: \"eac94ec2-102e-481e-a744-c17f25a49374\") " pod="kuadrant-system/authorino-7bcfd8f9cd-44hmv"
Sep 09 15:33:22.023167 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:22.023100    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-8rbfz" event={"ID":"4e149c93-225f-4e52-95f8-3377915cb90d","Type":"ContainerStarted","Data":"ef7ab116ff5cfdb644daf9e3592d900312f523cff67f1e648c46e3bcda50fc92"}
Sep 09 15:33:22.067501 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:22.067477    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:33:22.107471 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:22.107441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx9dc\" (UniqueName: \"kubernetes.io/projected/eac94ec2-102e-481e-a744-c17f25a49374-kube-api-access-rx9dc\") pod \"authorino-7bcfd8f9cd-44hmv\" (UID: \"eac94ec2-102e-481e-a744-c17f25a49374\") " pod="kuadrant-system/authorino-7bcfd8f9cd-44hmv"
Sep 09 15:33:22.115388 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:22.115366    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx9dc\" (UniqueName: \"kubernetes.io/projected/eac94ec2-102e-481e-a744-c17f25a49374-kube-api-access-rx9dc\") pod \"authorino-7bcfd8f9cd-44hmv\" (UID: \"eac94ec2-102e-481e-a744-c17f25a49374\") " pod="kuadrant-system/authorino-7bcfd8f9cd-44hmv"
Sep 09 15:33:22.289050 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:22.288954    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-44hmv"
Sep 09 15:33:22.555905 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:22.555829    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-44hmv"]
Sep 09 15:33:22.558531 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:33:22.558500    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeac94ec2_102e_481e_a744_c17f25a49374.slice/crio-48fea7d17cfa2a4133a8f54281ac4a2e1215012d5308bfced8ce6cd29e262ac6 WatchSource:0}: Error finding container 48fea7d17cfa2a4133a8f54281ac4a2e1215012d5308bfced8ce6cd29e262ac6: Status 404 returned error can't find the container with id 48fea7d17cfa2a4133a8f54281ac4a2e1215012d5308bfced8ce6cd29e262ac6
Sep 09 15:33:23.029847 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:23.029813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-8rbfz" event={"ID":"4e149c93-225f-4e52-95f8-3377915cb90d","Type":"ContainerStarted","Data":"3fcf9bebfd0e6b84bc14de82b8ce6ab5ed5309b8e77e0eab17ad855c7b05fbfa"}
Sep 09 15:33:23.030039 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:23.029949    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-8rbfz"
Sep 09 15:33:23.030945 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:23.030924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-44hmv" event={"ID":"eac94ec2-102e-481e-a744-c17f25a49374","Type":"ContainerStarted","Data":"48fea7d17cfa2a4133a8f54281ac4a2e1215012d5308bfced8ce6cd29e262ac6"}
Sep 09 15:33:23.044793 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:23.044748    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-8rbfz" podStartSLOduration=0.871670997 podStartE2EDuration="2.044732539s" totalImagesPullingTime="1.173061542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:33:23.044447549 +0000 UTC m=+792.686489539" watchObservedRunningTime="2026-09-09 15:33:23.044732539 +0000 UTC m=+792.686774530"
Sep 09 15:33:25.042243 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:25.042201    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-44hmv" event={"ID":"eac94ec2-102e-481e-a744-c17f25a49374","Type":"ContainerStarted","Data":"6e3c46a5c383372bd48dc6005001bb5483ba0c3aea0d9b53b9bf09d191fc72fd"}
Sep 09 15:33:25.057144 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:25.057102    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-44hmv" podStartSLOduration=2.431333796 podStartE2EDuration="4.057087066s" totalImagesPullingTime="1.62575327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:33:25.055896208 +0000 UTC m=+794.697938198" watchObservedRunningTime="2026-09-09 15:33:25.057087066 +0000 UTC m=+794.699129056"
Sep 09 15:33:25.705216 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:25.705187    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-44hmv"]
Sep 09 15:33:27.051699 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:27.051659    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-44hmv" podUID="eac94ec2-102e-481e-a744-c17f25a49374" containerName="authorino" containerID="cri-o://6e3c46a5c383372bd48dc6005001bb5483ba0c3aea0d9b53b9bf09d191fc72fd" gracePeriod=30
Sep 09 15:33:27.296393 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:27.296369    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-44hmv"
Sep 09 15:33:27.347771 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:27.347694    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx9dc\" (UniqueName: \"kubernetes.io/projected/eac94ec2-102e-481e-a744-c17f25a49374-kube-api-access-rx9dc\") pod \"eac94ec2-102e-481e-a744-c17f25a49374\" (UID: \"eac94ec2-102e-481e-a744-c17f25a49374\") "
Sep 09 15:33:27.349857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:27.349836    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac94ec2-102e-481e-a744-c17f25a49374-kube-api-access-rx9dc" (OuterVolumeSpecName: "kube-api-access-rx9dc") pod "eac94ec2-102e-481e-a744-c17f25a49374" (UID: "eac94ec2-102e-481e-a744-c17f25a49374"). InnerVolumeSpecName "kube-api-access-rx9dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:33:27.448811 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:27.448780    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx9dc\" (UniqueName: \"kubernetes.io/projected/eac94ec2-102e-481e-a744-c17f25a49374-kube-api-access-rx9dc\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:33:28.058188 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:28.058158    2578 generic.go:356] "Generic (PLEG): container finished" podID="eac94ec2-102e-481e-a744-c17f25a49374" containerID="6e3c46a5c383372bd48dc6005001bb5483ba0c3aea0d9b53b9bf09d191fc72fd" exitCode=0
Sep 09 15:33:28.058587 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:28.058204    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-44hmv"
Sep 09 15:33:28.058587 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:28.058220    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-44hmv" event={"ID":"eac94ec2-102e-481e-a744-c17f25a49374","Type":"ContainerDied","Data":"6e3c46a5c383372bd48dc6005001bb5483ba0c3aea0d9b53b9bf09d191fc72fd"}
Sep 09 15:33:28.058587 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:28.058265    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-44hmv" event={"ID":"eac94ec2-102e-481e-a744-c17f25a49374","Type":"ContainerDied","Data":"48fea7d17cfa2a4133a8f54281ac4a2e1215012d5308bfced8ce6cd29e262ac6"}
Sep 09 15:33:28.058587 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:28.058283    2578 scope.go:117] "RemoveContainer" containerID="6e3c46a5c383372bd48dc6005001bb5483ba0c3aea0d9b53b9bf09d191fc72fd"
Sep 09 15:33:28.067333 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:28.067315    2578 scope.go:117] "RemoveContainer" containerID="6e3c46a5c383372bd48dc6005001bb5483ba0c3aea0d9b53b9bf09d191fc72fd"
Sep 09 15:33:28.067632 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:33:28.067608    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e3c46a5c383372bd48dc6005001bb5483ba0c3aea0d9b53b9bf09d191fc72fd\": container with ID starting with 6e3c46a5c383372bd48dc6005001bb5483ba0c3aea0d9b53b9bf09d191fc72fd not found: ID does not exist" containerID="6e3c46a5c383372bd48dc6005001bb5483ba0c3aea0d9b53b9bf09d191fc72fd"
Sep 09 15:33:28.067694 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:28.067634    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e3c46a5c383372bd48dc6005001bb5483ba0c3aea0d9b53b9bf09d191fc72fd"} err="failed to get container status \"6e3c46a5c383372bd48dc6005001bb5483ba0c3aea0d9b53b9bf09d191fc72fd\": rpc error: code = NotFound desc = could not find container \"6e3c46a5c383372bd48dc6005001bb5483ba0c3aea0d9b53b9bf09d191fc72fd\": container with ID starting with 6e3c46a5c383372bd48dc6005001bb5483ba0c3aea0d9b53b9bf09d191fc72fd not found: ID does not exist"
Sep 09 15:33:28.095821 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:28.095799    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-44hmv"]
Sep 09 15:33:28.100379 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:28.100359    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-44hmv"]
Sep 09 15:33:28.866564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:28.866535    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eac94ec2-102e-481e-a744-c17f25a49374" path="/var/lib/kubelet/pods/eac94ec2-102e-481e-a744-c17f25a49374/volumes"
Sep 09 15:33:34.037044 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:34.036946    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-8rbfz"
Sep 09 15:33:35.753239 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:35.753205    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8rbfz"]
Sep 09 15:33:35.753678 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:35.753375    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-8rbfz" podUID="4e149c93-225f-4e52-95f8-3377915cb90d" containerName="limitador" containerID="cri-o://3fcf9bebfd0e6b84bc14de82b8ce6ab5ed5309b8e77e0eab17ad855c7b05fbfa" gracePeriod=30
Sep 09 15:33:36.299654 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:36.299632    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-8rbfz"
Sep 09 15:33:36.326668 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:36.326605    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4e149c93-225f-4e52-95f8-3377915cb90d-config-file\") pod \"4e149c93-225f-4e52-95f8-3377915cb90d\" (UID: \"4e149c93-225f-4e52-95f8-3377915cb90d\") "
Sep 09 15:33:36.326798 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:36.326711    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv8m6\" (UniqueName: \"kubernetes.io/projected/4e149c93-225f-4e52-95f8-3377915cb90d-kube-api-access-dv8m6\") pod \"4e149c93-225f-4e52-95f8-3377915cb90d\" (UID: \"4e149c93-225f-4e52-95f8-3377915cb90d\") "
Sep 09 15:33:36.327027 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:36.326973    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e149c93-225f-4e52-95f8-3377915cb90d-config-file" (OuterVolumeSpecName: "config-file") pod "4e149c93-225f-4e52-95f8-3377915cb90d" (UID: "4e149c93-225f-4e52-95f8-3377915cb90d"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:33:36.328920 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:36.328893    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e149c93-225f-4e52-95f8-3377915cb90d-kube-api-access-dv8m6" (OuterVolumeSpecName: "kube-api-access-dv8m6") pod "4e149c93-225f-4e52-95f8-3377915cb90d" (UID: "4e149c93-225f-4e52-95f8-3377915cb90d"). InnerVolumeSpecName "kube-api-access-dv8m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:33:36.428158 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:36.428129    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv8m6\" (UniqueName: \"kubernetes.io/projected/4e149c93-225f-4e52-95f8-3377915cb90d-kube-api-access-dv8m6\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:33:36.428158 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:36.428153    2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4e149c93-225f-4e52-95f8-3377915cb90d-config-file\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:33:37.100488 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:37.100451    2578 generic.go:356] "Generic (PLEG): container finished" podID="4e149c93-225f-4e52-95f8-3377915cb90d" containerID="3fcf9bebfd0e6b84bc14de82b8ce6ab5ed5309b8e77e0eab17ad855c7b05fbfa" exitCode=0
Sep 09 15:33:37.100868 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:37.100513    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-8rbfz"
Sep 09 15:33:37.100868 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:37.100534    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-8rbfz" event={"ID":"4e149c93-225f-4e52-95f8-3377915cb90d","Type":"ContainerDied","Data":"3fcf9bebfd0e6b84bc14de82b8ce6ab5ed5309b8e77e0eab17ad855c7b05fbfa"}
Sep 09 15:33:37.100868 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:37.100602    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-8rbfz" event={"ID":"4e149c93-225f-4e52-95f8-3377915cb90d","Type":"ContainerDied","Data":"ef7ab116ff5cfdb644daf9e3592d900312f523cff67f1e648c46e3bcda50fc92"}
Sep 09 15:33:37.100868 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:37.100624    2578 scope.go:117] "RemoveContainer" containerID="3fcf9bebfd0e6b84bc14de82b8ce6ab5ed5309b8e77e0eab17ad855c7b05fbfa"
Sep 09 15:33:37.108957 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:37.108939    2578 scope.go:117] "RemoveContainer" containerID="3fcf9bebfd0e6b84bc14de82b8ce6ab5ed5309b8e77e0eab17ad855c7b05fbfa"
Sep 09 15:33:37.109270 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:33:37.109251    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fcf9bebfd0e6b84bc14de82b8ce6ab5ed5309b8e77e0eab17ad855c7b05fbfa\": container with ID starting with 3fcf9bebfd0e6b84bc14de82b8ce6ab5ed5309b8e77e0eab17ad855c7b05fbfa not found: ID does not exist" containerID="3fcf9bebfd0e6b84bc14de82b8ce6ab5ed5309b8e77e0eab17ad855c7b05fbfa"
Sep 09 15:33:37.109320 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:37.109278    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fcf9bebfd0e6b84bc14de82b8ce6ab5ed5309b8e77e0eab17ad855c7b05fbfa"} err="failed to get container status \"3fcf9bebfd0e6b84bc14de82b8ce6ab5ed5309b8e77e0eab17ad855c7b05fbfa\": rpc error: code = NotFound desc = could not find container \"3fcf9bebfd0e6b84bc14de82b8ce6ab5ed5309b8e77e0eab17ad855c7b05fbfa\": container with ID starting with 3fcf9bebfd0e6b84bc14de82b8ce6ab5ed5309b8e77e0eab17ad855c7b05fbfa not found: ID does not exist"
Sep 09 15:33:37.115396 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:37.115373    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8rbfz"]
Sep 09 15:33:37.118957 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:37.118937    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8rbfz"]
Sep 09 15:33:38.493705 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:38.493673    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:33:38.594190 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:38.594158    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:33:38.867905 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:38.867833    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e149c93-225f-4e52-95f8-3377915cb90d" path="/var/lib/kubelet/pods/4e149c93-225f-4e52-95f8-3377915cb90d/volumes"
Sep 09 15:33:38.941621 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:38.941588    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qcp97"]
Sep 09 15:33:38.942179 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:38.942164    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eac94ec2-102e-481e-a744-c17f25a49374" containerName="authorino"
Sep 09 15:33:38.942226 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:38.942184    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eac94ec2-102e-481e-a744-c17f25a49374" containerName="authorino"
Sep 09 15:33:38.942226 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:38.942199    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e149c93-225f-4e52-95f8-3377915cb90d" containerName="limitador"
Sep 09 15:33:38.942226 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:38.942208    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e149c93-225f-4e52-95f8-3377915cb90d" containerName="limitador"
Sep 09 15:33:38.942326 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:38.942286    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eac94ec2-102e-481e-a744-c17f25a49374" containerName="authorino"
Sep 09 15:33:38.942326 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:38.942297    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e149c93-225f-4e52-95f8-3377915cb90d" containerName="limitador"
Sep 09 15:33:38.951930 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:38.951908    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qcp97"]
Sep 09 15:33:38.952070 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:38.952050    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcp97"
Sep 09 15:33:38.954479 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:38.954458    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 15:33:38.954610 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:38.954496    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 09 15:33:38.955387 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:38.955366    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 15:33:38.955516 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:38.955498    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-4f74t\""
Sep 09 15:33:38.973028 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:38.973007    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:33:39.051228 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:39.051200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b94dd558-d97c-430b-a20b-494b4d89e3fa-data\") pod \"postgres-69fb4594cf-qcp97\" (UID: \"b94dd558-d97c-430b-a20b-494b4d89e3fa\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcp97"
Sep 09 15:33:39.051399 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:39.051240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f76rv\" (UniqueName: \"kubernetes.io/projected/b94dd558-d97c-430b-a20b-494b4d89e3fa-kube-api-access-f76rv\") pod \"postgres-69fb4594cf-qcp97\" (UID: \"b94dd558-d97c-430b-a20b-494b4d89e3fa\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcp97"
Sep 09 15:33:39.151919 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:39.151883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f76rv\" (UniqueName: \"kubernetes.io/projected/b94dd558-d97c-430b-a20b-494b4d89e3fa-kube-api-access-f76rv\") pod \"postgres-69fb4594cf-qcp97\" (UID: \"b94dd558-d97c-430b-a20b-494b4d89e3fa\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcp97"
Sep 09 15:33:39.152123 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:39.152038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b94dd558-d97c-430b-a20b-494b4d89e3fa-data\") pod \"postgres-69fb4594cf-qcp97\" (UID: \"b94dd558-d97c-430b-a20b-494b4d89e3fa\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcp97"
Sep 09 15:33:39.152413 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:39.152396    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b94dd558-d97c-430b-a20b-494b4d89e3fa-data\") pod \"postgres-69fb4594cf-qcp97\" (UID: \"b94dd558-d97c-430b-a20b-494b4d89e3fa\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcp97"
Sep 09 15:33:39.160377 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:39.160357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f76rv\" (UniqueName: \"kubernetes.io/projected/b94dd558-d97c-430b-a20b-494b4d89e3fa-kube-api-access-f76rv\") pod \"postgres-69fb4594cf-qcp97\" (UID: \"b94dd558-d97c-430b-a20b-494b4d89e3fa\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcp97"
Sep 09 15:33:39.263679 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:39.263651    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcp97"
Sep 09 15:33:39.534901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:39.534873    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qcp97"]
Sep 09 15:33:39.537459 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:33:39.537432    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94dd558_d97c_430b_a20b_494b4d89e3fa.slice/crio-11b9c9c698c6f415223aab50951807767e201a3a08c2b6c1201420ef8dc00a09 WatchSource:0}: Error finding container 11b9c9c698c6f415223aab50951807767e201a3a08c2b6c1201420ef8dc00a09: Status 404 returned error can't find the container with id 11b9c9c698c6f415223aab50951807767e201a3a08c2b6c1201420ef8dc00a09
Sep 09 15:33:40.117474 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:40.117444    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcp97" event={"ID":"b94dd558-d97c-430b-a20b-494b4d89e3fa","Type":"ContainerStarted","Data":"11b9c9c698c6f415223aab50951807767e201a3a08c2b6c1201420ef8dc00a09"}
Sep 09 15:33:45.145872 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:45.145836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcp97" event={"ID":"b94dd558-d97c-430b-a20b-494b4d89e3fa","Type":"ContainerStarted","Data":"b1dc8a66ec98d1a9c1bfaad1d5dd039a18ae3a07b667c931252814da87b3a96b"}
Sep 09 15:33:45.146297 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:45.145911    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcp97"
Sep 09 15:33:45.161155 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:45.161114    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcp97" podStartSLOduration=2.371437994 podStartE2EDuration="7.161100603s" totalImagesPullingTime="4.789662609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:33:45.160019404 +0000 UTC m=+814.802061394" watchObservedRunningTime="2026-09-09 15:33:45.161100603 +0000 UTC m=+814.803142592"
Sep 09 15:33:51.179677 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.179648    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcp97"
Sep 09 15:33:51.483154 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.483072    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:33:51.706857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.706821    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:33:51.707217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.707199    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-vpb9m"]
Sep 09 15:33:51.710500 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.710485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-vpb9m"
Sep 09 15:33:51.712865 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.712665    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9frgw\""
Sep 09 15:33:51.714749 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.714725    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-vpb9m"]
Sep 09 15:33:51.750388 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.750323    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-vpb9m"]
Sep 09 15:33:51.750579 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:33:51.750562    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-nn5qr], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-vpb9m" podUID="32f7be7a-0e7f-4ea5-9cf9-6611732ad252"
Sep 09 15:33:51.765151 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.765126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn5qr\" (UniqueName: \"kubernetes.io/projected/32f7be7a-0e7f-4ea5-9cf9-6611732ad252-kube-api-access-nn5qr\") pod \"authorino-557f876f69-vpb9m\" (UID: \"32f7be7a-0e7f-4ea5-9cf9-6611732ad252\") " pod="kuadrant-system/authorino-557f876f69-vpb9m"
Sep 09 15:33:51.772256 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.772233    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-hk5x9"]
Sep 09 15:33:51.775512 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.775498    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-hk5x9"
Sep 09 15:33:51.777612 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.777592    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 15:33:51.782824 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.782802    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-hk5x9"]
Sep 09 15:33:51.866173 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.866144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/12c5417c-d2f3-431a-ab70-4b24ab2363c0-tls-cert\") pod \"authorino-6d4dfc649f-hk5x9\" (UID: \"12c5417c-d2f3-431a-ab70-4b24ab2363c0\") " pod="kuadrant-system/authorino-6d4dfc649f-hk5x9"
Sep 09 15:33:51.866343 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.866191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5xpv\" (UniqueName: \"kubernetes.io/projected/12c5417c-d2f3-431a-ab70-4b24ab2363c0-kube-api-access-z5xpv\") pod \"authorino-6d4dfc649f-hk5x9\" (UID: \"12c5417c-d2f3-431a-ab70-4b24ab2363c0\") " pod="kuadrant-system/authorino-6d4dfc649f-hk5x9"
Sep 09 15:33:51.866343 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.866217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn5qr\" (UniqueName: \"kubernetes.io/projected/32f7be7a-0e7f-4ea5-9cf9-6611732ad252-kube-api-access-nn5qr\") pod \"authorino-557f876f69-vpb9m\" (UID: \"32f7be7a-0e7f-4ea5-9cf9-6611732ad252\") " pod="kuadrant-system/authorino-557f876f69-vpb9m"
Sep 09 15:33:51.874126 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.874097    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn5qr\" (UniqueName: \"kubernetes.io/projected/32f7be7a-0e7f-4ea5-9cf9-6611732ad252-kube-api-access-nn5qr\") pod \"authorino-557f876f69-vpb9m\" (UID: \"32f7be7a-0e7f-4ea5-9cf9-6611732ad252\") " pod="kuadrant-system/authorino-557f876f69-vpb9m"
Sep 09 15:33:51.946944 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.946907    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-hk5x9"]
Sep 09 15:33:51.947204 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:33:51.947185    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-z5xpv tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-hk5x9" podUID="12c5417c-d2f3-431a-ab70-4b24ab2363c0"
Sep 09 15:33:51.966780 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.966756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/12c5417c-d2f3-431a-ab70-4b24ab2363c0-tls-cert\") pod \"authorino-6d4dfc649f-hk5x9\" (UID: \"12c5417c-d2f3-431a-ab70-4b24ab2363c0\") " pod="kuadrant-system/authorino-6d4dfc649f-hk5x9"
Sep 09 15:33:51.966942 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.966798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5xpv\" (UniqueName: \"kubernetes.io/projected/12c5417c-d2f3-431a-ab70-4b24ab2363c0-kube-api-access-z5xpv\") pod \"authorino-6d4dfc649f-hk5x9\" (UID: \"12c5417c-d2f3-431a-ab70-4b24ab2363c0\") " pod="kuadrant-system/authorino-6d4dfc649f-hk5x9"
Sep 09 15:33:51.968225 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.968204    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c69c49fd-mkf6r"]
Sep 09 15:33:51.975686 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.975658    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/12c5417c-d2f3-431a-ab70-4b24ab2363c0-tls-cert\") pod \"authorino-6d4dfc649f-hk5x9\" (UID: \"12c5417c-d2f3-431a-ab70-4b24ab2363c0\") " pod="kuadrant-system/authorino-6d4dfc649f-hk5x9"
Sep 09 15:33:51.978810 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.978788    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c69c49fd-mkf6r"
Sep 09 15:33:51.980094 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.980070    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c69c49fd-mkf6r"]
Sep 09 15:33:51.982414 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:51.982394    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5xpv\" (UniqueName: \"kubernetes.io/projected/12c5417c-d2f3-431a-ab70-4b24ab2363c0-kube-api-access-z5xpv\") pod \"authorino-6d4dfc649f-hk5x9\" (UID: \"12c5417c-d2f3-431a-ab70-4b24ab2363c0\") " pod="kuadrant-system/authorino-6d4dfc649f-hk5x9"
Sep 09 15:33:52.068164 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.068132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88dd907d-7850-4f91-9dbd-bceb368f39ad-tls-cert\") pod \"authorino-5c69c49fd-mkf6r\" (UID: \"88dd907d-7850-4f91-9dbd-bceb368f39ad\") " pod="kuadrant-system/authorino-5c69c49fd-mkf6r"
Sep 09 15:33:52.068319 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.068192    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrnkv\" (UniqueName: \"kubernetes.io/projected/88dd907d-7850-4f91-9dbd-bceb368f39ad-kube-api-access-wrnkv\") pod \"authorino-5c69c49fd-mkf6r\" (UID: \"88dd907d-7850-4f91-9dbd-bceb368f39ad\") " pod="kuadrant-system/authorino-5c69c49fd-mkf6r"
Sep 09 15:33:52.169489 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.169454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrnkv\" (UniqueName: \"kubernetes.io/projected/88dd907d-7850-4f91-9dbd-bceb368f39ad-kube-api-access-wrnkv\") pod \"authorino-5c69c49fd-mkf6r\" (UID: \"88dd907d-7850-4f91-9dbd-bceb368f39ad\") " pod="kuadrant-system/authorino-5c69c49fd-mkf6r"
Sep 09 15:33:52.169626 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.169540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88dd907d-7850-4f91-9dbd-bceb368f39ad-tls-cert\") pod \"authorino-5c69c49fd-mkf6r\" (UID: \"88dd907d-7850-4f91-9dbd-bceb368f39ad\") " pod="kuadrant-system/authorino-5c69c49fd-mkf6r"
Sep 09 15:33:52.172137 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.172118    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88dd907d-7850-4f91-9dbd-bceb368f39ad-tls-cert\") pod \"authorino-5c69c49fd-mkf6r\" (UID: \"88dd907d-7850-4f91-9dbd-bceb368f39ad\") " pod="kuadrant-system/authorino-5c69c49fd-mkf6r"
Sep 09 15:33:52.176089 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.176058    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-vpb9m"
Sep 09 15:33:52.176187 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.176058    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-hk5x9"
Sep 09 15:33:52.177761 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.177737    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrnkv\" (UniqueName: \"kubernetes.io/projected/88dd907d-7850-4f91-9dbd-bceb368f39ad-kube-api-access-wrnkv\") pod \"authorino-5c69c49fd-mkf6r\" (UID: \"88dd907d-7850-4f91-9dbd-bceb368f39ad\") " pod="kuadrant-system/authorino-5c69c49fd-mkf6r"
Sep 09 15:33:52.205765 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.205733    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-vpb9m"
Sep 09 15:33:52.208697 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.208680    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-hk5x9"
Sep 09 15:33:52.270330 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.270271    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/12c5417c-d2f3-431a-ab70-4b24ab2363c0-tls-cert\") pod \"12c5417c-d2f3-431a-ab70-4b24ab2363c0\" (UID: \"12c5417c-d2f3-431a-ab70-4b24ab2363c0\") "
Sep 09 15:33:52.270461 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.270367    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nn5qr\" (UniqueName: \"kubernetes.io/projected/32f7be7a-0e7f-4ea5-9cf9-6611732ad252-kube-api-access-nn5qr\") pod \"32f7be7a-0e7f-4ea5-9cf9-6611732ad252\" (UID: \"32f7be7a-0e7f-4ea5-9cf9-6611732ad252\") "
Sep 09 15:33:52.270461 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.270419    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5xpv\" (UniqueName: \"kubernetes.io/projected/12c5417c-d2f3-431a-ab70-4b24ab2363c0-kube-api-access-z5xpv\") pod \"12c5417c-d2f3-431a-ab70-4b24ab2363c0\" (UID: \"12c5417c-d2f3-431a-ab70-4b24ab2363c0\") "
Sep 09 15:33:52.272528 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.272505    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c5417c-d2f3-431a-ab70-4b24ab2363c0-kube-api-access-z5xpv" (OuterVolumeSpecName: "kube-api-access-z5xpv") pod "12c5417c-d2f3-431a-ab70-4b24ab2363c0" (UID: "12c5417c-d2f3-431a-ab70-4b24ab2363c0"). InnerVolumeSpecName "kube-api-access-z5xpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:33:52.272528 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.272514    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f7be7a-0e7f-4ea5-9cf9-6611732ad252-kube-api-access-nn5qr" (OuterVolumeSpecName: "kube-api-access-nn5qr") pod "32f7be7a-0e7f-4ea5-9cf9-6611732ad252" (UID: "32f7be7a-0e7f-4ea5-9cf9-6611732ad252"). InnerVolumeSpecName "kube-api-access-nn5qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:33:52.272528 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.272522    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c5417c-d2f3-431a-ab70-4b24ab2363c0-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "12c5417c-d2f3-431a-ab70-4b24ab2363c0" (UID: "12c5417c-d2f3-431a-ab70-4b24ab2363c0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:33:52.307697 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.307677    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c69c49fd-mkf6r"
Sep 09 15:33:52.371243 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.371209    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5xpv\" (UniqueName: \"kubernetes.io/projected/12c5417c-d2f3-431a-ab70-4b24ab2363c0-kube-api-access-z5xpv\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:33:52.371243 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.371244    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/12c5417c-d2f3-431a-ab70-4b24ab2363c0-tls-cert\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:33:52.371421 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.371259    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn5qr\" (UniqueName: \"kubernetes.io/projected/32f7be7a-0e7f-4ea5-9cf9-6611732ad252-kube-api-access-nn5qr\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:33:52.584856 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:33:52.584820    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88dd907d_7850_4f91_9dbd_bceb368f39ad.slice/crio-1fd27dc9bb8205c666d83e52064c9057d23a96aae62d86a585ce53d6181319f4 WatchSource:0}: Error finding container 1fd27dc9bb8205c666d83e52064c9057d23a96aae62d86a585ce53d6181319f4: Status 404 returned error can't find the container with id 1fd27dc9bb8205c666d83e52064c9057d23a96aae62d86a585ce53d6181319f4
Sep 09 15:33:52.585786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:52.585763    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c69c49fd-mkf6r"]
Sep 09 15:33:53.181443 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:53.181417    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-hk5x9"
Sep 09 15:33:53.181600 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:53.181451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c69c49fd-mkf6r" event={"ID":"88dd907d-7850-4f91-9dbd-bceb368f39ad","Type":"ContainerStarted","Data":"1fd27dc9bb8205c666d83e52064c9057d23a96aae62d86a585ce53d6181319f4"}
Sep 09 15:33:53.181753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:53.181629    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-vpb9m"
Sep 09 15:33:53.207469 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:53.207437    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-hk5x9"]
Sep 09 15:33:53.210102 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:53.210082    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-hk5x9"]
Sep 09 15:33:53.228516 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:53.228487    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-vpb9m"]
Sep 09 15:33:53.232222 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:53.232196    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-vpb9m"]
Sep 09 15:33:54.866684 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:54.866656    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12c5417c-d2f3-431a-ab70-4b24ab2363c0" path="/var/lib/kubelet/pods/12c5417c-d2f3-431a-ab70-4b24ab2363c0/volumes"
Sep 09 15:33:54.867085 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:54.866871    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32f7be7a-0e7f-4ea5-9cf9-6611732ad252" path="/var/lib/kubelet/pods/32f7be7a-0e7f-4ea5-9cf9-6611732ad252/volumes"
Sep 09 15:33:57.205016 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:57.204961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c69c49fd-mkf6r" event={"ID":"88dd907d-7850-4f91-9dbd-bceb368f39ad","Type":"ContainerStarted","Data":"99d506fe85d1ebb2dafb4d1cde9aae95ab38daa650f69922083c4a5ddb3128d0"}
Sep 09 15:33:57.219110 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:33:57.219062    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5c69c49fd-mkf6r" podStartSLOduration=2.097536455 podStartE2EDuration="6.219047979s" totalImagesPullingTime="4.121511524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:33:57.218169999 +0000 UTC m=+826.860211988" watchObservedRunningTime="2026-09-09 15:33:57.219047979 +0000 UTC m=+826.861089969"
Sep 09 15:34:03.003339 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.003306    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:34:03.024067 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.024037    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59cc4f6cb6-qsdnh"]
Sep 09 15:34:03.027793 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.027772    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh"
Sep 09 15:34:03.030331 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.030305    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 15:34:03.030425 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.030306    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 15:34:03.030425 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.030306    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 15:34:03.030425 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.030315    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bsjmm\""
Sep 09 15:34:03.039644 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.039617    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59cc4f6cb6-qsdnh"]
Sep 09 15:34:03.103243 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.103206    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:34:03.167902 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.167867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc0090eb-bfed-4f1c-93eb-f44435e35a91-metrics-certs\") pod \"maas-controller-59cc4f6cb6-qsdnh\" (UID: \"fc0090eb-bfed-4f1c-93eb-f44435e35a91\") " pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh"
Sep 09 15:34:03.168111 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.167930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qf7g\" (UniqueName: \"kubernetes.io/projected/fc0090eb-bfed-4f1c-93eb-f44435e35a91-kube-api-access-2qf7g\") pod \"maas-controller-59cc4f6cb6-qsdnh\" (UID: \"fc0090eb-bfed-4f1c-93eb-f44435e35a91\") " pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh"
Sep 09 15:34:03.168111 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.168058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc0090eb-bfed-4f1c-93eb-f44435e35a91-webhook-cert\") pod \"maas-controller-59cc4f6cb6-qsdnh\" (UID: \"fc0090eb-bfed-4f1c-93eb-f44435e35a91\") " pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh"
Sep 09 15:34:03.202552 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.202518    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:34:03.209663 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.209638    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f6f9bc65f-sw57k"]
Sep 09 15:34:03.213020 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.213004    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f9bc65f-sw57k"
Sep 09 15:34:03.222768 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.222744    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f6f9bc65f-sw57k"]
Sep 09 15:34:03.266578 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.266508    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:34:03.268504 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.268481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc0090eb-bfed-4f1c-93eb-f44435e35a91-webhook-cert\") pod \"maas-controller-59cc4f6cb6-qsdnh\" (UID: \"fc0090eb-bfed-4f1c-93eb-f44435e35a91\") " pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh"
Sep 09 15:34:03.268618 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.268535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc0090eb-bfed-4f1c-93eb-f44435e35a91-metrics-certs\") pod \"maas-controller-59cc4f6cb6-qsdnh\" (UID: \"fc0090eb-bfed-4f1c-93eb-f44435e35a91\") " pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh"
Sep 09 15:34:03.268618 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.268588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qf7g\" (UniqueName: \"kubernetes.io/projected/fc0090eb-bfed-4f1c-93eb-f44435e35a91-kube-api-access-2qf7g\") pod \"maas-controller-59cc4f6cb6-qsdnh\" (UID: \"fc0090eb-bfed-4f1c-93eb-f44435e35a91\") " pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh"
Sep 09 15:34:03.271250 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.271222    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc0090eb-bfed-4f1c-93eb-f44435e35a91-webhook-cert\") pod \"maas-controller-59cc4f6cb6-qsdnh\" (UID: \"fc0090eb-bfed-4f1c-93eb-f44435e35a91\") " pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh"
Sep 09 15:34:03.271352 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.271255    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc0090eb-bfed-4f1c-93eb-f44435e35a91-metrics-certs\") pod \"maas-controller-59cc4f6cb6-qsdnh\" (UID: \"fc0090eb-bfed-4f1c-93eb-f44435e35a91\") " pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh"
Sep 09 15:34:03.276269 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.276248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qf7g\" (UniqueName: \"kubernetes.io/projected/fc0090eb-bfed-4f1c-93eb-f44435e35a91-kube-api-access-2qf7g\") pod \"maas-controller-59cc4f6cb6-qsdnh\" (UID: \"fc0090eb-bfed-4f1c-93eb-f44435e35a91\") " pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh"
Sep 09 15:34:03.338399 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.338368    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh"
Sep 09 15:34:03.369733 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.369700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93263f27-3bc8-4744-b5d2-91fd622adf08-webhook-cert\") pod \"maas-controller-7f6f9bc65f-sw57k\" (UID: \"93263f27-3bc8-4744-b5d2-91fd622adf08\") " pod="opendatahub/maas-controller-7f6f9bc65f-sw57k"
Sep 09 15:34:03.369900 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.369746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nmkk\" (UniqueName: \"kubernetes.io/projected/93263f27-3bc8-4744-b5d2-91fd622adf08-kube-api-access-4nmkk\") pod \"maas-controller-7f6f9bc65f-sw57k\" (UID: \"93263f27-3bc8-4744-b5d2-91fd622adf08\") " pod="opendatahub/maas-controller-7f6f9bc65f-sw57k"
Sep 09 15:34:03.369900 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.369865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93263f27-3bc8-4744-b5d2-91fd622adf08-metrics-certs\") pod \"maas-controller-7f6f9bc65f-sw57k\" (UID: \"93263f27-3bc8-4744-b5d2-91fd622adf08\") " pod="opendatahub/maas-controller-7f6f9bc65f-sw57k"
Sep 09 15:34:03.470623 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.470589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93263f27-3bc8-4744-b5d2-91fd622adf08-webhook-cert\") pod \"maas-controller-7f6f9bc65f-sw57k\" (UID: \"93263f27-3bc8-4744-b5d2-91fd622adf08\") " pod="opendatahub/maas-controller-7f6f9bc65f-sw57k"
Sep 09 15:34:03.470623 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.470625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nmkk\" (UniqueName: \"kubernetes.io/projected/93263f27-3bc8-4744-b5d2-91fd622adf08-kube-api-access-4nmkk\") pod \"maas-controller-7f6f9bc65f-sw57k\" (UID: \"93263f27-3bc8-4744-b5d2-91fd622adf08\") " pod="opendatahub/maas-controller-7f6f9bc65f-sw57k"
Sep 09 15:34:03.470828 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.470680    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93263f27-3bc8-4744-b5d2-91fd622adf08-metrics-certs\") pod \"maas-controller-7f6f9bc65f-sw57k\" (UID: \"93263f27-3bc8-4744-b5d2-91fd622adf08\") " pod="opendatahub/maas-controller-7f6f9bc65f-sw57k"
Sep 09 15:34:03.473225 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.473199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93263f27-3bc8-4744-b5d2-91fd622adf08-metrics-certs\") pod \"maas-controller-7f6f9bc65f-sw57k\" (UID: \"93263f27-3bc8-4744-b5d2-91fd622adf08\") " pod="opendatahub/maas-controller-7f6f9bc65f-sw57k"
Sep 09 15:34:03.473344 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.473271    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93263f27-3bc8-4744-b5d2-91fd622adf08-webhook-cert\") pod \"maas-controller-7f6f9bc65f-sw57k\" (UID: \"93263f27-3bc8-4744-b5d2-91fd622adf08\") " pod="opendatahub/maas-controller-7f6f9bc65f-sw57k"
Sep 09 15:34:03.479132 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.479115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nmkk\" (UniqueName: \"kubernetes.io/projected/93263f27-3bc8-4744-b5d2-91fd622adf08-kube-api-access-4nmkk\") pod \"maas-controller-7f6f9bc65f-sw57k\" (UID: \"93263f27-3bc8-4744-b5d2-91fd622adf08\") " pod="opendatahub/maas-controller-7f6f9bc65f-sw57k"
Sep 09 15:34:03.523744 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.523664    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f9bc65f-sw57k"
Sep 09 15:34:03.610259 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.610230    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59cc4f6cb6-qsdnh"]
Sep 09 15:34:03.612150 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:34:03.612126    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0090eb_bfed_4f1c_93eb_f44435e35a91.slice/crio-fc8f0b60a18328304e1e1539f9e2a260e59f4a4a5e13ae73c7cbeed346f888a7 WatchSource:0}: Error finding container fc8f0b60a18328304e1e1539f9e2a260e59f4a4a5e13ae73c7cbeed346f888a7: Status 404 returned error can't find the container with id fc8f0b60a18328304e1e1539f9e2a260e59f4a4a5e13ae73c7cbeed346f888a7
Sep 09 15:34:03.792974 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:03.792950    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f6f9bc65f-sw57k"]
Sep 09 15:34:03.794615 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:34:03.794592    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93263f27_3bc8_4744_b5d2_91fd622adf08.slice/crio-e2f90b8358484b1eb4efcb7d4149e44cf507ef1ef3e0fe082bdce0e69771dbed WatchSource:0}: Error finding container e2f90b8358484b1eb4efcb7d4149e44cf507ef1ef3e0fe082bdce0e69771dbed: Status 404 returned error can't find the container with id e2f90b8358484b1eb4efcb7d4149e44cf507ef1ef3e0fe082bdce0e69771dbed
Sep 09 15:34:04.242155 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:04.242114    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f9bc65f-sw57k" event={"ID":"93263f27-3bc8-4744-b5d2-91fd622adf08","Type":"ContainerStarted","Data":"e2f90b8358484b1eb4efcb7d4149e44cf507ef1ef3e0fe082bdce0e69771dbed"}
Sep 09 15:34:04.243788 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:04.243758    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh" event={"ID":"fc0090eb-bfed-4f1c-93eb-f44435e35a91","Type":"ContainerStarted","Data":"fc8f0b60a18328304e1e1539f9e2a260e59f4a4a5e13ae73c7cbeed346f888a7"}
Sep 09 15:34:06.468254 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:06.468222    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:34:06.570641 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:06.570544    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:34:07.263025 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:07.262971    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f9bc65f-sw57k" event={"ID":"93263f27-3bc8-4744-b5d2-91fd622adf08","Type":"ContainerStarted","Data":"715eaaa9f5458a6a29113a7f8197d72ab3ca4b95477a4c93a9cd25bbee8e571e"}
Sep 09 15:34:07.263209 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:07.263115    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f6f9bc65f-sw57k"
Sep 09 15:34:07.264498 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:07.264477    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh" event={"ID":"fc0090eb-bfed-4f1c-93eb-f44435e35a91","Type":"ContainerStarted","Data":"65e12547a58adac4bfc0b19317a7ae31247c67d9a6df847883564d56e71c6f62"}
Sep 09 15:34:07.264612 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:07.264550    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh"
Sep 09 15:34:07.280691 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:07.280646    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f6f9bc65f-sw57k" podStartSLOduration=1.81137064 podStartE2EDuration="4.280633267s" totalImagesPullingTime="2.469262627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:34:07.278899607 +0000 UTC m=+836.920941596" watchObservedRunningTime="2026-09-09 15:34:07.280633267 +0000 UTC m=+836.922675255"
Sep 09 15:34:07.294646 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:07.294606    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh" podStartSLOduration=1.656408442 podStartE2EDuration="4.294593246s" totalImagesPullingTime="2.638184804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:34:07.293294002 +0000 UTC m=+836.935336002" watchObservedRunningTime="2026-09-09 15:34:07.294593246 +0000 UTC m=+836.936635234"
Sep 09 15:34:18.271212 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:18.271185    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh"
Sep 09 15:34:18.271597 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:18.271253    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f6f9bc65f-sw57k"
Sep 09 15:34:18.329468 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:18.329439    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59cc4f6cb6-qsdnh"]
Sep 09 15:34:18.329708 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:18.329682    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh" podUID="fc0090eb-bfed-4f1c-93eb-f44435e35a91" containerName="manager" containerID="cri-o://65e12547a58adac4bfc0b19317a7ae31247c67d9a6df847883564d56e71c6f62" gracePeriod=30
Sep 09 15:34:18.579019 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:18.578990    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh"
Sep 09 15:34:18.604431 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:18.604404    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc0090eb-bfed-4f1c-93eb-f44435e35a91-webhook-cert\") pod \"fc0090eb-bfed-4f1c-93eb-f44435e35a91\" (UID: \"fc0090eb-bfed-4f1c-93eb-f44435e35a91\") "
Sep 09 15:34:18.604600 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:18.604449    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc0090eb-bfed-4f1c-93eb-f44435e35a91-metrics-certs\") pod \"fc0090eb-bfed-4f1c-93eb-f44435e35a91\" (UID: \"fc0090eb-bfed-4f1c-93eb-f44435e35a91\") "
Sep 09 15:34:18.604600 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:18.604487    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qf7g\" (UniqueName: \"kubernetes.io/projected/fc0090eb-bfed-4f1c-93eb-f44435e35a91-kube-api-access-2qf7g\") pod \"fc0090eb-bfed-4f1c-93eb-f44435e35a91\" (UID: \"fc0090eb-bfed-4f1c-93eb-f44435e35a91\") "
Sep 09 15:34:18.606830 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:18.606789    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0090eb-bfed-4f1c-93eb-f44435e35a91-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fc0090eb-bfed-4f1c-93eb-f44435e35a91" (UID: "fc0090eb-bfed-4f1c-93eb-f44435e35a91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:34:18.606956 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:18.606835    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0090eb-bfed-4f1c-93eb-f44435e35a91-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fc0090eb-bfed-4f1c-93eb-f44435e35a91" (UID: "fc0090eb-bfed-4f1c-93eb-f44435e35a91"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:34:18.606956 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:18.606882    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0090eb-bfed-4f1c-93eb-f44435e35a91-kube-api-access-2qf7g" (OuterVolumeSpecName: "kube-api-access-2qf7g") pod "fc0090eb-bfed-4f1c-93eb-f44435e35a91" (UID: "fc0090eb-bfed-4f1c-93eb-f44435e35a91"). InnerVolumeSpecName "kube-api-access-2qf7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:34:18.705431 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:18.705400    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qf7g\" (UniqueName: \"kubernetes.io/projected/fc0090eb-bfed-4f1c-93eb-f44435e35a91-kube-api-access-2qf7g\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:34:18.705431 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:18.705426    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc0090eb-bfed-4f1c-93eb-f44435e35a91-webhook-cert\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:34:18.705431 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:18.705434    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc0090eb-bfed-4f1c-93eb-f44435e35a91-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:34:19.322067 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:19.322024    2578 generic.go:356] "Generic (PLEG): container finished" podID="fc0090eb-bfed-4f1c-93eb-f44435e35a91" containerID="65e12547a58adac4bfc0b19317a7ae31247c67d9a6df847883564d56e71c6f62" exitCode=0
Sep 09 15:34:19.322508 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:19.322082    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh"
Sep 09 15:34:19.322508 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:19.322084    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh" event={"ID":"fc0090eb-bfed-4f1c-93eb-f44435e35a91","Type":"ContainerDied","Data":"65e12547a58adac4bfc0b19317a7ae31247c67d9a6df847883564d56e71c6f62"}
Sep 09 15:34:19.322508 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:19.322217    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59cc4f6cb6-qsdnh" event={"ID":"fc0090eb-bfed-4f1c-93eb-f44435e35a91","Type":"ContainerDied","Data":"fc8f0b60a18328304e1e1539f9e2a260e59f4a4a5e13ae73c7cbeed346f888a7"}
Sep 09 15:34:19.322508 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:19.322248    2578 scope.go:117] "RemoveContainer" containerID="65e12547a58adac4bfc0b19317a7ae31247c67d9a6df847883564d56e71c6f62"
Sep 09 15:34:19.332178 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:19.331959    2578 scope.go:117] "RemoveContainer" containerID="65e12547a58adac4bfc0b19317a7ae31247c67d9a6df847883564d56e71c6f62"
Sep 09 15:34:19.332289 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:34:19.332267    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e12547a58adac4bfc0b19317a7ae31247c67d9a6df847883564d56e71c6f62\": container with ID starting with 65e12547a58adac4bfc0b19317a7ae31247c67d9a6df847883564d56e71c6f62 not found: ID does not exist" containerID="65e12547a58adac4bfc0b19317a7ae31247c67d9a6df847883564d56e71c6f62"
Sep 09 15:34:19.332332 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:19.332297    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e12547a58adac4bfc0b19317a7ae31247c67d9a6df847883564d56e71c6f62"} err="failed to get container status \"65e12547a58adac4bfc0b19317a7ae31247c67d9a6df847883564d56e71c6f62\": rpc error: code = NotFound desc = could not find container \"65e12547a58adac4bfc0b19317a7ae31247c67d9a6df847883564d56e71c6f62\": container with ID starting with 65e12547a58adac4bfc0b19317a7ae31247c67d9a6df847883564d56e71c6f62 not found: ID does not exist"
Sep 09 15:34:19.339380 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:19.339357    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59cc4f6cb6-qsdnh"]
Sep 09 15:34:19.343245 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:19.343224    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-59cc4f6cb6-qsdnh"]
Sep 09 15:34:20.867216 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:20.867185    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc0090eb-bfed-4f1c-93eb-f44435e35a91" path="/var/lib/kubelet/pods/fc0090eb-bfed-4f1c-93eb-f44435e35a91/volumes"
Sep 09 15:34:39.069433 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:39.069396    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:34:39.169349 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:39.169302    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:34:40.498312 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.498276    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:34:40.599532 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.599505    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:34:40.648050 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.648018    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"]
Sep 09 15:34:40.648388 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.648376    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc0090eb-bfed-4f1c-93eb-f44435e35a91" containerName="manager"
Sep 09 15:34:40.648433 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.648390    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc0090eb-bfed-4f1c-93eb-f44435e35a91" containerName="manager"
Sep 09 15:34:40.648484 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.648475    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc0090eb-bfed-4f1c-93eb-f44435e35a91" containerName="manager"
Sep 09 15:34:40.652737 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.652724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"
Sep 09 15:34:40.655563 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.655534    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lw9zj\""
Sep 09 15:34:40.655699 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.655643    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 15:34:40.655699 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.655643    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 15:34:40.662809 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.662787    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"]
Sep 09 15:34:40.794730 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.794647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jphwb\" (UniqueName: \"kubernetes.io/projected/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-kube-api-access-jphwb\") pod \"maas-api-6d864d7888-wltcf\" (UID: \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\") " pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"
Sep 09 15:34:40.794730 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.794709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-metrics-certs\") pod \"maas-api-6d864d7888-wltcf\" (UID: \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\") " pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"
Sep 09 15:34:40.794917 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.794800    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-maas-api-tls\") pod \"maas-api-6d864d7888-wltcf\" (UID: \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\") " pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"
Sep 09 15:34:40.794917 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.794843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-openshift-service-ca\") pod \"maas-api-6d864d7888-wltcf\" (UID: \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\") " pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"
Sep 09 15:34:40.895988 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.895955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-metrics-certs\") pod \"maas-api-6d864d7888-wltcf\" (UID: \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\") " pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"
Sep 09 15:34:40.896197 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.896036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-maas-api-tls\") pod \"maas-api-6d864d7888-wltcf\" (UID: \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\") " pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"
Sep 09 15:34:40.896197 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.896084    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-openshift-service-ca\") pod \"maas-api-6d864d7888-wltcf\" (UID: \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\") " pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"
Sep 09 15:34:40.896197 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.896128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jphwb\" (UniqueName: \"kubernetes.io/projected/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-kube-api-access-jphwb\") pod \"maas-api-6d864d7888-wltcf\" (UID: \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\") " pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"
Sep 09 15:34:40.896736 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.896714    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-openshift-service-ca\") pod \"maas-api-6d864d7888-wltcf\" (UID: \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\") " pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"
Sep 09 15:34:40.898487 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.898464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-maas-api-tls\") pod \"maas-api-6d864d7888-wltcf\" (UID: \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\") " pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"
Sep 09 15:34:40.898694 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.898677    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-metrics-certs\") pod \"maas-api-6d864d7888-wltcf\" (UID: \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\") " pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"
Sep 09 15:34:40.907458 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.907437    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jphwb\" (UniqueName: \"kubernetes.io/projected/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-kube-api-access-jphwb\") pod \"maas-api-6d864d7888-wltcf\" (UID: \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\") " pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"
Sep 09 15:34:40.907871 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.907853    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:34:40.962306 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:40.962279    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"
Sep 09 15:34:41.008783 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:41.008728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:34:41.113295 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:41.113219    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:34:41.195072 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:41.195042    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:34:41.249355 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:41.249317    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-qf4s4"]
Sep 09 15:34:41.252517 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:41.252495    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-qf4s4"
Sep 09 15:34:41.256017 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:41.255841    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 15:34:41.256017 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:41.256017    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-c5xnn\""
Sep 09 15:34:41.257556 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:41.256839    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"]
Sep 09 15:34:41.261118 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:34:41.261097    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6ab1fe_334f_42dd_8b08_fe3b91701a10.slice/crio-04fc6baaaa603178fbea6481ad1c6aa7a5a69858a0800c03ddf5b2f4333d7edf WatchSource:0}: Error finding container 04fc6baaaa603178fbea6481ad1c6aa7a5a69858a0800c03ddf5b2f4333d7edf: Status 404 returned error can't find the container with id 04fc6baaaa603178fbea6481ad1c6aa7a5a69858a0800c03ddf5b2f4333d7edf
Sep 09 15:34:41.266576 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:41.266552    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-qf4s4"]
Sep 09 15:34:41.300052 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:41.300009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c743a297-9127-4f66-8153-8a00d9327040-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-qf4s4\" (UID: \"c743a297-9127-4f66-8153-8a00d9327040\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-qf4s4"
Sep 09 15:34:41.300192 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:41.300093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cznz\" (UniqueName: \"kubernetes.io/projected/c743a297-9127-4f66-8153-8a00d9327040-kube-api-access-9cznz\") pod \"payload-processing-5b54b7fbb7-qf4s4\" (UID: \"c743a297-9127-4f66-8153-8a00d9327040\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-qf4s4"
Sep 09 15:34:41.400701 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:41.400663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c743a297-9127-4f66-8153-8a00d9327040-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-qf4s4\" (UID: \"c743a297-9127-4f66-8153-8a00d9327040\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-qf4s4"
Sep 09 15:34:41.400959 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:41.400931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cznz\" (UniqueName: \"kubernetes.io/projected/c743a297-9127-4f66-8153-8a00d9327040-kube-api-access-9cznz\") pod \"payload-processing-5b54b7fbb7-qf4s4\" (UID: \"c743a297-9127-4f66-8153-8a00d9327040\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-qf4s4"
Sep 09 15:34:41.401719 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:41.401685    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c743a297-9127-4f66-8153-8a00d9327040-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-qf4s4\" (UID: \"c743a297-9127-4f66-8153-8a00d9327040\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-qf4s4"
Sep 09 15:34:41.409515 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:41.409490    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cznz\" (UniqueName: \"kubernetes.io/projected/c743a297-9127-4f66-8153-8a00d9327040-kube-api-access-9cznz\") pod \"payload-processing-5b54b7fbb7-qf4s4\" (UID: \"c743a297-9127-4f66-8153-8a00d9327040\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-qf4s4"
Sep 09 15:34:41.434751 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:41.434708    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf" event={"ID":"0f6ab1fe-334f-42dd-8b08-fe3b91701a10","Type":"ContainerStarted","Data":"04fc6baaaa603178fbea6481ad1c6aa7a5a69858a0800c03ddf5b2f4333d7edf"}
Sep 09 15:34:41.568415 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:41.568379    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-qf4s4"
Sep 09 15:34:41.844873 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:41.844848    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-qf4s4"]
Sep 09 15:34:41.846629 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:34:41.846596    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc743a297_9127_4f66_8153_8a00d9327040.slice/crio-d7c26fa016a5a82811e0b91563195a170a4bbbcf66c502a0aec38e8807595365 WatchSource:0}: Error finding container d7c26fa016a5a82811e0b91563195a170a4bbbcf66c502a0aec38e8807595365: Status 404 returned error can't find the container with id d7c26fa016a5a82811e0b91563195a170a4bbbcf66c502a0aec38e8807595365
Sep 09 15:34:42.444847 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:42.444810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-qf4s4" event={"ID":"c743a297-9127-4f66-8153-8a00d9327040","Type":"ContainerStarted","Data":"d7c26fa016a5a82811e0b91563195a170a4bbbcf66c502a0aec38e8807595365"}
Sep 09 15:34:44.456451 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:44.456378    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf" event={"ID":"0f6ab1fe-334f-42dd-8b08-fe3b91701a10","Type":"ContainerStarted","Data":"319537013a699f14925adae4c159c1363a8ec64f5683309bfd8d55038f44ddb5"}
Sep 09 15:34:44.457018 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:44.456538    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"
Sep 09 15:34:44.474007 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:44.473941    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf" podStartSLOduration=2.29242855 podStartE2EDuration="4.473926008s" totalImagesPullingTime="2.181497458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:34:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:34:44.472685997 +0000 UTC m=+874.114728007" watchObservedRunningTime="2026-09-09 15:34:44.473926008 +0000 UTC m=+874.115968049"
Sep 09 15:34:44.898142 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:34:44.898105    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc743a297_9127_4f66_8153_8a00d9327040.slice/crio-920c6633a0b3a9969f7e48998aa09255eeda0eaa2a46040e2e2c272d8f851a91.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 15:34:45.463779 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:45.463745    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-qf4s4" event={"ID":"c743a297-9127-4f66-8153-8a00d9327040","Type":"ContainerStarted","Data":"920c6633a0b3a9969f7e48998aa09255eeda0eaa2a46040e2e2c272d8f851a91"}
Sep 09 15:34:45.464290 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:45.463930    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-qf4s4"
Sep 09 15:34:45.480478 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:45.480429    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-qf4s4" podStartSLOduration=1.48514546 podStartE2EDuration="4.480414486s" totalImagesPullingTime="2.995269026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:34:45.478567926 +0000 UTC m=+875.120609935" watchObservedRunningTime="2026-09-09 15:34:45.480414486 +0000 UTC m=+875.122456474"
Sep 09 15:34:50.469272 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:50.469242    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"
Sep 09 15:34:51.497629 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:51.497595    2578 generic.go:356] "Generic (PLEG): container finished" podID="d41c51d4-c973-4e22-a0a1-96c523e020f3" containerID="93615f450ef1a07089dca273f978e0c139bb6a43c2009ff7cedb851f129f9faf" exitCode=0
Sep 09 15:34:51.498045 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:51.497657    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m" event={"ID":"d41c51d4-c973-4e22-a0a1-96c523e020f3","Type":"ContainerDied","Data":"93615f450ef1a07089dca273f978e0c139bb6a43c2009ff7cedb851f129f9faf"}
Sep 09 15:34:51.498096 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:51.498072    2578 scope.go:117] "RemoveContainer" containerID="93615f450ef1a07089dca273f978e0c139bb6a43c2009ff7cedb851f129f9faf"
Sep 09 15:34:51.499027 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:51.499001    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 15:34:51.499093 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:51.499067    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 15:34:51.499133 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:51.499095    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 15:34:51.573724 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:51.573698    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:34:51.955043 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:51.954996    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:34:52.515103 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:52.515061    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m" event={"ID":"d41c51d4-c973-4e22-a0a1-96c523e020f3","Type":"ContainerStarted","Data":"d3b3c8b90007427f7979c539816cf85c80c637a8038f1a0322edf3b89500e65a"}
Sep 09 15:34:53.141457 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:53.141418    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:34:53.142576 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:53.142549    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-c784m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 09 15:34:53.142719 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:53.142609    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m" podUID="d41c51d4-c973-4e22-a0a1-96c523e020f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 09 15:34:53.248966 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:53.248933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:34:54.142507 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:54.142468    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-c784m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 09 15:34:54.142901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:54.142527    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m" podUID="d41c51d4-c973-4e22-a0a1-96c523e020f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 09 15:34:55.142436 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:55.142400    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-c784m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 09 15:34:55.142624 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:55.142456    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m" podUID="d41c51d4-c973-4e22-a0a1-96c523e020f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 09 15:34:56.142029 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:56.141967    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-c784m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 09 15:34:56.142208 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:56.142061    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m" podUID="d41c51d4-c973-4e22-a0a1-96c523e020f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 09 15:34:56.471725 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:56.471653    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-qf4s4"
Sep 09 15:34:57.141449 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:57.141410    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-c784m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 09 15:34:57.141601 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:57.141481    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m" podUID="d41c51d4-c973-4e22-a0a1-96c523e020f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 09 15:34:58.145302 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:58.145273    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:34:58.145725 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:58.145654    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:34:58.146263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:34:58.146247    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-c784m"
Sep 09 15:35:07.988847 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:07.988775    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:35:14.965204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:14.965165    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:35:15.103860 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:15.103834    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 15:35:15.104446 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:15.104422    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 15:35:15.111881 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:15.111862    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:35:15.112334 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:15.112311    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:35:26.711509 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:26.711468    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:35:33.659481 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.659442    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"]
Sep 09 15:35:33.666496 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.666473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.670137 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.670106    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 15:35:33.670137 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.670124    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"]
Sep 09 15:35:33.670311 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.670147    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 15:35:33.670311 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.670106    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 09 15:35:33.670311 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.670109    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dz98x\""
Sep 09 15:35:33.704179 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.704138    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:35:33.782727 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.782694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d4615a-378a-4737-9f51-723397da6166-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2zsxj\" (UID: \"d3d4615a-378a-4737-9f51-723397da6166\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.782877 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.782739    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn9fg\" (UniqueName: \"kubernetes.io/projected/d3d4615a-378a-4737-9f51-723397da6166-kube-api-access-rn9fg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2zsxj\" (UID: \"d3d4615a-378a-4737-9f51-723397da6166\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.782877 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.782802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3d4615a-378a-4737-9f51-723397da6166-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2zsxj\" (UID: \"d3d4615a-378a-4737-9f51-723397da6166\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.782877 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.782819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d4615a-378a-4737-9f51-723397da6166-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2zsxj\" (UID: \"d3d4615a-378a-4737-9f51-723397da6166\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.783027 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.782919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d4615a-378a-4737-9f51-723397da6166-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2zsxj\" (UID: \"d3d4615a-378a-4737-9f51-723397da6166\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.783027 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.782959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d4615a-378a-4737-9f51-723397da6166-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2zsxj\" (UID: \"d3d4615a-378a-4737-9f51-723397da6166\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.883816 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.883778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn9fg\" (UniqueName: \"kubernetes.io/projected/d3d4615a-378a-4737-9f51-723397da6166-kube-api-access-rn9fg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2zsxj\" (UID: \"d3d4615a-378a-4737-9f51-723397da6166\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.884019 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.883837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3d4615a-378a-4737-9f51-723397da6166-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2zsxj\" (UID: \"d3d4615a-378a-4737-9f51-723397da6166\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.884019 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.883864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d4615a-378a-4737-9f51-723397da6166-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2zsxj\" (UID: \"d3d4615a-378a-4737-9f51-723397da6166\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.884019 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.883924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d4615a-378a-4737-9f51-723397da6166-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2zsxj\" (UID: \"d3d4615a-378a-4737-9f51-723397da6166\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.884019 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.883968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d4615a-378a-4737-9f51-723397da6166-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2zsxj\" (UID: \"d3d4615a-378a-4737-9f51-723397da6166\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.884211 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.884035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d4615a-378a-4737-9f51-723397da6166-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2zsxj\" (UID: \"d3d4615a-378a-4737-9f51-723397da6166\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.884328 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.884306    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3d4615a-378a-4737-9f51-723397da6166-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2zsxj\" (UID: \"d3d4615a-378a-4737-9f51-723397da6166\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.884396 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.884379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3d4615a-378a-4737-9f51-723397da6166-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2zsxj\" (UID: \"d3d4615a-378a-4737-9f51-723397da6166\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.884455 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.884397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3d4615a-378a-4737-9f51-723397da6166-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2zsxj\" (UID: \"d3d4615a-378a-4737-9f51-723397da6166\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.886465 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.886436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3d4615a-378a-4737-9f51-723397da6166-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2zsxj\" (UID: \"d3d4615a-378a-4737-9f51-723397da6166\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.886577 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.886566    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3d4615a-378a-4737-9f51-723397da6166-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2zsxj\" (UID: \"d3d4615a-378a-4737-9f51-723397da6166\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.892168 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.892146    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn9fg\" (UniqueName: \"kubernetes.io/projected/d3d4615a-378a-4737-9f51-723397da6166-kube-api-access-rn9fg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2zsxj\" (UID: \"d3d4615a-378a-4737-9f51-723397da6166\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:33.980175 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:33.980103    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:34.256611 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:34.256587    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"]
Sep 09 15:35:34.259311 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:35:34.259279    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d4615a_378a_4737_9f51_723397da6166.slice/crio-ea2c2bedb22ba099fbb22a8ede92837dd7c28ece5b963bc8e69a78ef986ebaed WatchSource:0}: Error finding container ea2c2bedb22ba099fbb22a8ede92837dd7c28ece5b963bc8e69a78ef986ebaed: Status 404 returned error can't find the container with id ea2c2bedb22ba099fbb22a8ede92837dd7c28ece5b963bc8e69a78ef986ebaed
Sep 09 15:35:34.727912 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:34.727862    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj" event={"ID":"d3d4615a-378a-4737-9f51-723397da6166","Type":"ContainerStarted","Data":"ea2c2bedb22ba099fbb22a8ede92837dd7c28ece5b963bc8e69a78ef986ebaed"}
Sep 09 15:35:35.734619 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:35.734578    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj" event={"ID":"d3d4615a-378a-4737-9f51-723397da6166","Type":"ContainerStarted","Data":"9703fc563cc13765e97d4f931af50457fbe6c676bcf56ad1923358a860b0e242"}
Sep 09 15:35:35.762584 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:35.762528    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj" podStartSLOduration=1.7169109310000001 podStartE2EDuration="2.762511791s" totalImagesPullingTime="1.04560086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:35:35.760821987 +0000 UTC m=+925.402863977" watchObservedRunningTime="2026-09-09 15:35:35.762511791 +0000 UTC m=+925.404553776"
Sep 09 15:35:43.980321 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:43.980289    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:43.980790 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:43.980355    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:43.993187 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:43.993154    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:44.790476 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:44.790448    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2zsxj"
Sep 09 15:35:47.188855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:47.188826    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:35:57.352242 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.352207    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"]
Sep 09 15:35:57.358739 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.358700    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.360953 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.360932    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 09 15:35:57.365422 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.365396    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"]
Sep 09 15:35:57.388678 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.388653    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:35:57.392739 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.392716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8d5797-6341-4a87-b2dd-81185f930a2e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjsm5\" (UID: \"0e8d5797-6341-4a87-b2dd-81185f930a2e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.392858 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.392762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gh2j\" (UniqueName: \"kubernetes.io/projected/0e8d5797-6341-4a87-b2dd-81185f930a2e-kube-api-access-4gh2j\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjsm5\" (UID: \"0e8d5797-6341-4a87-b2dd-81185f930a2e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.392858 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.392838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8d5797-6341-4a87-b2dd-81185f930a2e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjsm5\" (UID: \"0e8d5797-6341-4a87-b2dd-81185f930a2e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.392971 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.392859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8d5797-6341-4a87-b2dd-81185f930a2e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjsm5\" (UID: \"0e8d5797-6341-4a87-b2dd-81185f930a2e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.392971 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.392944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8d5797-6341-4a87-b2dd-81185f930a2e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjsm5\" (UID: \"0e8d5797-6341-4a87-b2dd-81185f930a2e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.393102 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.393030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8d5797-6341-4a87-b2dd-81185f930a2e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjsm5\" (UID: \"0e8d5797-6341-4a87-b2dd-81185f930a2e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.493868 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.493833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8d5797-6341-4a87-b2dd-81185f930a2e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjsm5\" (UID: \"0e8d5797-6341-4a87-b2dd-81185f930a2e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.494049 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.493889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8d5797-6341-4a87-b2dd-81185f930a2e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjsm5\" (UID: \"0e8d5797-6341-4a87-b2dd-81185f930a2e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.494049 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.493955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8d5797-6341-4a87-b2dd-81185f930a2e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjsm5\" (UID: \"0e8d5797-6341-4a87-b2dd-81185f930a2e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.494049 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.494007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gh2j\" (UniqueName: \"kubernetes.io/projected/0e8d5797-6341-4a87-b2dd-81185f930a2e-kube-api-access-4gh2j\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjsm5\" (UID: \"0e8d5797-6341-4a87-b2dd-81185f930a2e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.494205 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.494048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8d5797-6341-4a87-b2dd-81185f930a2e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjsm5\" (UID: \"0e8d5797-6341-4a87-b2dd-81185f930a2e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.494205 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.494071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8d5797-6341-4a87-b2dd-81185f930a2e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjsm5\" (UID: \"0e8d5797-6341-4a87-b2dd-81185f930a2e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.494348 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.494322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8d5797-6341-4a87-b2dd-81185f930a2e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjsm5\" (UID: \"0e8d5797-6341-4a87-b2dd-81185f930a2e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.494431 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.494369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8d5797-6341-4a87-b2dd-81185f930a2e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjsm5\" (UID: \"0e8d5797-6341-4a87-b2dd-81185f930a2e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.494481 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.494466    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8d5797-6341-4a87-b2dd-81185f930a2e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjsm5\" (UID: \"0e8d5797-6341-4a87-b2dd-81185f930a2e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.496382 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.496353    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8d5797-6341-4a87-b2dd-81185f930a2e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjsm5\" (UID: \"0e8d5797-6341-4a87-b2dd-81185f930a2e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.496649 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.496630    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8d5797-6341-4a87-b2dd-81185f930a2e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjsm5\" (UID: \"0e8d5797-6341-4a87-b2dd-81185f930a2e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.502751 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.502727    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gh2j\" (UniqueName: \"kubernetes.io/projected/0e8d5797-6341-4a87-b2dd-81185f930a2e-kube-api-access-4gh2j\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pjsm5\" (UID: \"0e8d5797-6341-4a87-b2dd-81185f930a2e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.671658 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.671620    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:35:57.942699 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:57.942672    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"]
Sep 09 15:35:57.944523 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:35:57.944495    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8d5797_6341_4a87_b2dd_81185f930a2e.slice/crio-a49a7a064a50e1551dc7d9679f75b30a9c4980f54259423a5b1f83e899315489 WatchSource:0}: Error finding container a49a7a064a50e1551dc7d9679f75b30a9c4980f54259423a5b1f83e899315489: Status 404 returned error can't find the container with id a49a7a064a50e1551dc7d9679f75b30a9c4980f54259423a5b1f83e899315489
Sep 09 15:35:58.850754 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:58.850717    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5" event={"ID":"0e8d5797-6341-4a87-b2dd-81185f930a2e","Type":"ContainerStarted","Data":"73de874ffb897452222852d6b3b4f9809c8e0d0bb97dea332764d3ba4c9e29f9"}
Sep 09 15:35:58.851168 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:58.850769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5" event={"ID":"0e8d5797-6341-4a87-b2dd-81185f930a2e","Type":"ContainerStarted","Data":"a49a7a064a50e1551dc7d9679f75b30a9c4980f54259423a5b1f83e899315489"}
Sep 09 15:35:58.868111 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:35:58.868066    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5" podStartSLOduration=1.636075386 podStartE2EDuration="1.868050623s" totalImagesPullingTime="231.975237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:35:58.866756607 +0000 UTC m=+948.508798598" watchObservedRunningTime="2026-09-09 15:35:58.868050623 +0000 UTC m=+948.510092612"
Sep 09 15:36:07.672338 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:36:07.672300    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:36:07.672769 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:36:07.672358    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:36:07.684709 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:36:07.684683    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:36:07.904848 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:36:07.904820    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pjsm5"
Sep 09 15:36:14.525810 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:36:14.525766    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:36:15.590729 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:36:15.590652    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:28.767814 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:28.767779    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c69c49fd-mkf6r"]
Sep 09 15:37:28.768275 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:28.768026    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5c69c49fd-mkf6r" podUID="88dd907d-7850-4f91-9dbd-bceb368f39ad" containerName="authorino" containerID="cri-o://99d506fe85d1ebb2dafb4d1cde9aae95ab38daa650f69922083c4a5ddb3128d0" gracePeriod=30
Sep 09 15:37:29.011632 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:29.011606    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c69c49fd-mkf6r"
Sep 09 15:37:29.129891 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:29.129783    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrnkv\" (UniqueName: \"kubernetes.io/projected/88dd907d-7850-4f91-9dbd-bceb368f39ad-kube-api-access-wrnkv\") pod \"88dd907d-7850-4f91-9dbd-bceb368f39ad\" (UID: \"88dd907d-7850-4f91-9dbd-bceb368f39ad\") "
Sep 09 15:37:29.129891 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:29.129836    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88dd907d-7850-4f91-9dbd-bceb368f39ad-tls-cert\") pod \"88dd907d-7850-4f91-9dbd-bceb368f39ad\" (UID: \"88dd907d-7850-4f91-9dbd-bceb368f39ad\") "
Sep 09 15:37:29.132157 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:29.132128    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88dd907d-7850-4f91-9dbd-bceb368f39ad-kube-api-access-wrnkv" (OuterVolumeSpecName: "kube-api-access-wrnkv") pod "88dd907d-7850-4f91-9dbd-bceb368f39ad" (UID: "88dd907d-7850-4f91-9dbd-bceb368f39ad"). InnerVolumeSpecName "kube-api-access-wrnkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:37:29.142564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:29.142534    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88dd907d-7850-4f91-9dbd-bceb368f39ad-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "88dd907d-7850-4f91-9dbd-bceb368f39ad" (UID: "88dd907d-7850-4f91-9dbd-bceb368f39ad"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:37:29.230562 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:29.230523    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrnkv\" (UniqueName: \"kubernetes.io/projected/88dd907d-7850-4f91-9dbd-bceb368f39ad-kube-api-access-wrnkv\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:37:29.230562 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:29.230554    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88dd907d-7850-4f91-9dbd-bceb368f39ad-tls-cert\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:37:29.283944 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:29.283912    2578 generic.go:356] "Generic (PLEG): container finished" podID="88dd907d-7850-4f91-9dbd-bceb368f39ad" containerID="99d506fe85d1ebb2dafb4d1cde9aae95ab38daa650f69922083c4a5ddb3128d0" exitCode=0
Sep 09 15:37:29.284138 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:29.283964    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c69c49fd-mkf6r"
Sep 09 15:37:29.284138 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:29.284027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c69c49fd-mkf6r" event={"ID":"88dd907d-7850-4f91-9dbd-bceb368f39ad","Type":"ContainerDied","Data":"99d506fe85d1ebb2dafb4d1cde9aae95ab38daa650f69922083c4a5ddb3128d0"}
Sep 09 15:37:29.284138 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:29.284070    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c69c49fd-mkf6r" event={"ID":"88dd907d-7850-4f91-9dbd-bceb368f39ad","Type":"ContainerDied","Data":"1fd27dc9bb8205c666d83e52064c9057d23a96aae62d86a585ce53d6181319f4"}
Sep 09 15:37:29.284138 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:29.284092    2578 scope.go:117] "RemoveContainer" containerID="99d506fe85d1ebb2dafb4d1cde9aae95ab38daa650f69922083c4a5ddb3128d0"
Sep 09 15:37:29.295521 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:29.295505    2578 scope.go:117] "RemoveContainer" containerID="99d506fe85d1ebb2dafb4d1cde9aae95ab38daa650f69922083c4a5ddb3128d0"
Sep 09 15:37:29.295800 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:37:29.295783    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d506fe85d1ebb2dafb4d1cde9aae95ab38daa650f69922083c4a5ddb3128d0\": container with ID starting with 99d506fe85d1ebb2dafb4d1cde9aae95ab38daa650f69922083c4a5ddb3128d0 not found: ID does not exist" containerID="99d506fe85d1ebb2dafb4d1cde9aae95ab38daa650f69922083c4a5ddb3128d0"
Sep 09 15:37:29.295876 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:29.295807    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d506fe85d1ebb2dafb4d1cde9aae95ab38daa650f69922083c4a5ddb3128d0"} err="failed to get container status \"99d506fe85d1ebb2dafb4d1cde9aae95ab38daa650f69922083c4a5ddb3128d0\": rpc error: code = NotFound desc = could not find container \"99d506fe85d1ebb2dafb4d1cde9aae95ab38daa650f69922083c4a5ddb3128d0\": container with ID starting with 99d506fe85d1ebb2dafb4d1cde9aae95ab38daa650f69922083c4a5ddb3128d0 not found: ID does not exist"
Sep 09 15:37:29.319764 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:29.319740    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c69c49fd-mkf6r"]
Sep 09 15:37:29.324412 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:29.324391    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5c69c49fd-mkf6r"]
Sep 09 15:37:29.328990 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:29.328960    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:29.422473 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:29.422448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:30.576386 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:30.576348    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:30.676420 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:30.676390    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:30.868147 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:30.868063    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88dd907d-7850-4f91-9dbd-bceb368f39ad" path="/var/lib/kubelet/pods/88dd907d-7850-4f91-9dbd-bceb368f39ad/volumes"
Sep 09 15:37:48.323542 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.323510    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:48.424600 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.424570    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:48.682687 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.682660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:48.729348 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.729319    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"]
Sep 09 15:37:48.729709 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.729697    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88dd907d-7850-4f91-9dbd-bceb368f39ad" containerName="authorino"
Sep 09 15:37:48.729754 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.729711    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88dd907d-7850-4f91-9dbd-bceb368f39ad" containerName="authorino"
Sep 09 15:37:48.730437 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.730400    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88dd907d-7850-4f91-9dbd-bceb368f39ad" containerName="authorino"
Sep 09 15:37:48.737074 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.735724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.739296 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.739267    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-57b579-openshift-default-dockercfg-lgd98\""
Sep 09 15:37:48.748354 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.748328    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"]
Sep 09 15:37:48.784168 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.784130    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:48.807085 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.807056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-podinfo\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.807246 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.807094    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-credential-socket\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.807246 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.807121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-envoy\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.807246 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.807202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-workload-socket\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.807246 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.807227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-data\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.807423 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.807277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-token\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.807423 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.807306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb883ca-8542-4127-b9be-2ba9571c8c93-istiod-ca-cert\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.807423 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.807349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-workload-certs\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.807423 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.807387    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsvpr\" (UniqueName: \"kubernetes.io/projected/5bb883ca-8542-4127-b9be-2ba9571c8c93-kube-api-access-jsvpr\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.864450 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.864410    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:48.908198 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.908165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-podinfo\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.908198 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.908199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-credential-socket\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.908430 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.908217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-envoy\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.908430 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.908249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-workload-socket\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.908430 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.908272    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-data\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.908430 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.908329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-token\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.908430 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.908348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb883ca-8542-4127-b9be-2ba9571c8c93-istiod-ca-cert\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.908430 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.908385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-workload-certs\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.908430 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.908425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvpr\" (UniqueName: \"kubernetes.io/projected/5bb883ca-8542-4127-b9be-2ba9571c8c93-kube-api-access-jsvpr\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.908789 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.908620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-credential-socket\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.908789 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.908688    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-workload-socket\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.908954 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.908932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-workload-certs\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.909220 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.909167    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-data\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.909220 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.909190    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb883ca-8542-4127-b9be-2ba9571c8c93-istiod-ca-cert\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.910739 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.910713    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-envoy\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.910940 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.910922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-podinfo\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.927035 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.927014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-token\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:48.927469 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:48.927447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsvpr\" (UniqueName: \"kubernetes.io/projected/5bb883ca-8542-4127-b9be-2ba9571c8c93-kube-api-access-jsvpr\") pod \"e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") " pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:49.049753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:49.049684    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:49.245871 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:49.245829    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:49.329580 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:49.329555    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"]
Sep 09 15:37:49.331107 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:49.331072    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:49.332079 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:37:49.332051    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb883ca_8542_4127_b9be_2ba9571c8c93.slice/crio-0ea61334c50bf29f87893ed979816e109d5e5fd3389eb4bcf4572da7a0002c09 WatchSource:0}: Error finding container 0ea61334c50bf29f87893ed979816e109d5e5fd3389eb4bcf4572da7a0002c09: Status 404 returned error can't find the container with id 0ea61334c50bf29f87893ed979816e109d5e5fd3389eb4bcf4572da7a0002c09
Sep 09 15:37:49.334077 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:49.334048    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 15:37:49.334179 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:49.334130    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 15:37:49.334238 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:49.334175    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 15:37:49.385941 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:49.385911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk" event={"ID":"5bb883ca-8542-4127-b9be-2ba9571c8c93","Type":"ContainerStarted","Data":"0ea61334c50bf29f87893ed979816e109d5e5fd3389eb4bcf4572da7a0002c09"}
Sep 09 15:37:49.965011 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:49.964951    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:50.064371 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:50.064328    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:50.392376 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:50.392342    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk" event={"ID":"5bb883ca-8542-4127-b9be-2ba9571c8c93","Type":"ContainerStarted","Data":"07613ebb5db0edf16932f290c59c23c0630f9c62e3a87b551fb0616e5f68637c"}
Sep 09 15:37:50.444736 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:50.444685    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk" podStartSLOduration=2.444667856 podStartE2EDuration="2.444667856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:37:50.443707544 +0000 UTC m=+1060.085749534" watchObservedRunningTime="2026-09-09 15:37:50.444667856 +0000 UTC m=+1060.086709845"
Sep 09 15:37:50.540621 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:50.540581    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:50.605811 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:50.605777    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7472-140647083564080"]
Sep 09 15:37:50.609306 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:50.609291    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-140647083564080"
Sep 09 15:37:50.611761 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:50.611735    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pcz7f\""
Sep 09 15:37:50.618239 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:50.618218    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7472-140647083564080"]
Sep 09 15:37:50.725459 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:50.725375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhmx4\" (UniqueName: \"kubernetes.io/projected/275ea03a-c215-441a-8a09-f23bdb9dee7a-kube-api-access-nhmx4\") pod \"test-curl-7472-140647083564080\" (UID: \"275ea03a-c215-441a-8a09-f23bdb9dee7a\") " pod="opendatahub/test-curl-7472-140647083564080"
Sep 09 15:37:50.826917 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:50.826858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhmx4\" (UniqueName: \"kubernetes.io/projected/275ea03a-c215-441a-8a09-f23bdb9dee7a-kube-api-access-nhmx4\") pod \"test-curl-7472-140647083564080\" (UID: \"275ea03a-c215-441a-8a09-f23bdb9dee7a\") " pod="opendatahub/test-curl-7472-140647083564080"
Sep 09 15:37:50.835037 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:50.835010    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhmx4\" (UniqueName: \"kubernetes.io/projected/275ea03a-c215-441a-8a09-f23bdb9dee7a-kube-api-access-nhmx4\") pod \"test-curl-7472-140647083564080\" (UID: \"275ea03a-c215-441a-8a09-f23bdb9dee7a\") " pod="opendatahub/test-curl-7472-140647083564080"
Sep 09 15:37:50.930404 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:50.930367    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-140647083564080"
Sep 09 15:37:51.050844 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:51.050816    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:51.056138 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:51.056112    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:51.201134 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:51.201112    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7472-140647083564080"]
Sep 09 15:37:51.203205 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:37:51.203180    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod275ea03a_c215_441a_8a09_f23bdb9dee7a.slice/crio-41e5db374fbb968f7fd0980f0443abd88e07bb07434998c0af4722883fe968d8 WatchSource:0}: Error finding container 41e5db374fbb968f7fd0980f0443abd88e07bb07434998c0af4722883fe968d8: Status 404 returned error can't find the container with id 41e5db374fbb968f7fd0980f0443abd88e07bb07434998c0af4722883fe968d8
Sep 09 15:37:51.204596 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:51.204580    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 15:37:51.398670 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:51.398639    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7472-140647083564080" event={"ID":"275ea03a-c215-441a-8a09-f23bdb9dee7a","Type":"ContainerStarted","Data":"41e5db374fbb968f7fd0980f0443abd88e07bb07434998c0af4722883fe968d8"}
Sep 09 15:37:51.399122 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:51.398904    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:51.399907 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:51.399886    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:37:52.031542 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:52.031506    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:52.130014 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:52.129968    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:52.232631 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:52.232605    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:52.405525 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:52.405494    2578 generic.go:356] "Generic (PLEG): container finished" podID="275ea03a-c215-441a-8a09-f23bdb9dee7a" containerID="d634cf831c528c3855ae9575b2977eb7e9de5dbc608065ea1e8280efaf17ba1f" exitCode=0
Sep 09 15:37:52.405964 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:52.405589    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7472-140647083564080" event={"ID":"275ea03a-c215-441a-8a09-f23bdb9dee7a","Type":"ContainerDied","Data":"d634cf831c528c3855ae9575b2977eb7e9de5dbc608065ea1e8280efaf17ba1f"}
Sep 09 15:37:52.473917 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:52.473880    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:52.574160 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:52.574127    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:53.153304 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.153277    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"]
Sep 09 15:37:53.156948 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.156928    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:37:53.159122 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.159089    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-df30be-serving-cert\""
Sep 09 15:37:53.167682 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.167645    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"]
Sep 09 15:37:53.199427 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.199401    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:53.250647 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.250605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb684ff-51f2-49c9-89e6-69fc847306cc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw\" (UID: \"7eb684ff-51f2-49c9-89e6-69fc847306cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:37:53.250647 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.250647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eb684ff-51f2-49c9-89e6-69fc847306cc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw\" (UID: \"7eb684ff-51f2-49c9-89e6-69fc847306cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:37:53.250838 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.250682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6gz9\" (UniqueName: \"kubernetes.io/projected/7eb684ff-51f2-49c9-89e6-69fc847306cc-kube-api-access-k6gz9\") pod \"maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw\" (UID: \"7eb684ff-51f2-49c9-89e6-69fc847306cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:37:53.250889 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.250846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb684ff-51f2-49c9-89e6-69fc847306cc-metrics-certs\") pod \"maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw\" (UID: \"7eb684ff-51f2-49c9-89e6-69fc847306cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:37:53.300179 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.300146    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:53.351409 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.351380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb684ff-51f2-49c9-89e6-69fc847306cc-metrics-certs\") pod \"maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw\" (UID: \"7eb684ff-51f2-49c9-89e6-69fc847306cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:37:53.351565 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.351432    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb684ff-51f2-49c9-89e6-69fc847306cc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw\" (UID: \"7eb684ff-51f2-49c9-89e6-69fc847306cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:37:53.351565 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.351452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eb684ff-51f2-49c9-89e6-69fc847306cc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw\" (UID: \"7eb684ff-51f2-49c9-89e6-69fc847306cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:37:53.351565 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.351482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6gz9\" (UniqueName: \"kubernetes.io/projected/7eb684ff-51f2-49c9-89e6-69fc847306cc-kube-api-access-k6gz9\") pod \"maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw\" (UID: \"7eb684ff-51f2-49c9-89e6-69fc847306cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:37:53.351565 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:37:53.351560    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-serving-cert: secret "maas-api-e2e-worker-w0-df30be-serving-cert" not found
Sep 09 15:37:53.351741 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:37:53.351629    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb684ff-51f2-49c9-89e6-69fc847306cc-maas-api-tls podName:7eb684ff-51f2-49c9-89e6-69fc847306cc nodeName:}" failed. No retries permitted until 2026-09-09 15:37:53.851610121 +0000 UTC m=+1063.493652088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7eb684ff-51f2-49c9-89e6-69fc847306cc-maas-api-tls") pod "maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw" (UID: "7eb684ff-51f2-49c9-89e6-69fc847306cc") : secret "maas-api-e2e-worker-w0-df30be-serving-cert" not found
Sep 09 15:37:53.352123 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.352101    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb684ff-51f2-49c9-89e6-69fc847306cc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw\" (UID: \"7eb684ff-51f2-49c9-89e6-69fc847306cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:37:53.354110 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.354092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb684ff-51f2-49c9-89e6-69fc847306cc-metrics-certs\") pod \"maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw\" (UID: \"7eb684ff-51f2-49c9-89e6-69fc847306cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:37:53.364300 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.364272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6gz9\" (UniqueName: \"kubernetes.io/projected/7eb684ff-51f2-49c9-89e6-69fc847306cc-kube-api-access-k6gz9\") pod \"maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw\" (UID: \"7eb684ff-51f2-49c9-89e6-69fc847306cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:37:53.458701 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.458672    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:53.547839 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.547815    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-140647083564080"
Sep 09 15:37:53.565924 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.565900    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7472-140647083564080_275ea03a-c215-441a-8a09-f23bdb9dee7a/test-curl-7472-140647083564080/0.log"
Sep 09 15:37:53.592774 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.592734    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7472-140647083564080"]
Sep 09 15:37:53.597472 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.597446    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7472-140647083564080"]
Sep 09 15:37:53.654672 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.654636    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhmx4\" (UniqueName: \"kubernetes.io/projected/275ea03a-c215-441a-8a09-f23bdb9dee7a-kube-api-access-nhmx4\") pod \"275ea03a-c215-441a-8a09-f23bdb9dee7a\" (UID: \"275ea03a-c215-441a-8a09-f23bdb9dee7a\") "
Sep 09 15:37:53.656879 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.656852    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275ea03a-c215-441a-8a09-f23bdb9dee7a-kube-api-access-nhmx4" (OuterVolumeSpecName: "kube-api-access-nhmx4") pod "275ea03a-c215-441a-8a09-f23bdb9dee7a" (UID: "275ea03a-c215-441a-8a09-f23bdb9dee7a"). InnerVolumeSpecName "kube-api-access-nhmx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:37:53.730182 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.730153    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7472-140647083561520"]
Sep 09 15:37:53.730597 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.730585    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="275ea03a-c215-441a-8a09-f23bdb9dee7a" containerName="test-curl-7472-140647083564080"
Sep 09 15:37:53.730641 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.730599    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="275ea03a-c215-441a-8a09-f23bdb9dee7a" containerName="test-curl-7472-140647083564080"
Sep 09 15:37:53.730683 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.730675    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="275ea03a-c215-441a-8a09-f23bdb9dee7a" containerName="test-curl-7472-140647083564080"
Sep 09 15:37:53.733915 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.733893    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-140647083561520"
Sep 09 15:37:53.741592 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.741572    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7472-140647083561520"]
Sep 09 15:37:53.756111 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.756085    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhmx4\" (UniqueName: \"kubernetes.io/projected/275ea03a-c215-441a-8a09-f23bdb9dee7a-kube-api-access-nhmx4\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:37:53.763100 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.763076    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:53.856700 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.856655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnh7q\" (UniqueName: \"kubernetes.io/projected/02e015d4-754d-4009-91d4-ac26f8af2f4e-kube-api-access-xnh7q\") pod \"test-curl-7472-140647083561520\" (UID: \"02e015d4-754d-4009-91d4-ac26f8af2f4e\") " pod="opendatahub/test-curl-7472-140647083561520"
Sep 09 15:37:53.856872 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.856809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eb684ff-51f2-49c9-89e6-69fc847306cc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw\" (UID: \"7eb684ff-51f2-49c9-89e6-69fc847306cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:37:53.859337 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.859286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eb684ff-51f2-49c9-89e6-69fc847306cc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw\" (UID: \"7eb684ff-51f2-49c9-89e6-69fc847306cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:37:53.958050 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.957938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnh7q\" (UniqueName: \"kubernetes.io/projected/02e015d4-754d-4009-91d4-ac26f8af2f4e-kube-api-access-xnh7q\") pod \"test-curl-7472-140647083561520\" (UID: \"02e015d4-754d-4009-91d4-ac26f8af2f4e\") " pod="opendatahub/test-curl-7472-140647083561520"
Sep 09 15:37:53.966532 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:53.966498    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnh7q\" (UniqueName: \"kubernetes.io/projected/02e015d4-754d-4009-91d4-ac26f8af2f4e-kube-api-access-xnh7q\") pod \"test-curl-7472-140647083561520\" (UID: \"02e015d4-754d-4009-91d4-ac26f8af2f4e\") " pod="opendatahub/test-curl-7472-140647083561520"
Sep 09 15:37:54.044831 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.044799    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-140647083561520"
Sep 09 15:37:54.069361 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.069334    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:37:54.253561 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.253474    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"]
Sep 09 15:37:54.259039 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.259016    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:37:54.261155 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.261136    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-1e15c8-serving-cert\""
Sep 09 15:37:54.264795 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.264770    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"]
Sep 09 15:37:54.319671 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.319645    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7472-140647083561520"]
Sep 09 15:37:54.321576 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:37:54.321554    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02e015d4_754d_4009_91d4_ac26f8af2f4e.slice/crio-ba58c0ffe44b5fe7f11035300b5c290d317ec4d15e01adf7d984fe2f5bd07047 WatchSource:0}: Error finding container ba58c0ffe44b5fe7f11035300b5c290d317ec4d15e01adf7d984fe2f5bd07047: Status 404 returned error can't find the container with id ba58c0ffe44b5fe7f11035300b5c290d317ec4d15e01adf7d984fe2f5bd07047
Sep 09 15:37:54.352660 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.352635    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"]
Sep 09 15:37:54.355960 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:37:54.355934    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb684ff_51f2_49c9_89e6_69fc847306cc.slice/crio-e4b0d3e2beb836de8d4985e7698ea47072ee1f99d1d42ff182c5dc71ec838032 WatchSource:0}: Error finding container e4b0d3e2beb836de8d4985e7698ea47072ee1f99d1d42ff182c5dc71ec838032: Status 404 returned error can't find the container with id e4b0d3e2beb836de8d4985e7698ea47072ee1f99d1d42ff182c5dc71ec838032
Sep 09 15:37:54.362548 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.362521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f63f69-2753-4d7d-9fd5-43534cc5db99-metrics-certs\") pod \"maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq\" (UID: \"03f63f69-2753-4d7d-9fd5-43534cc5db99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:37:54.362643 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.362598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f63f69-2753-4d7d-9fd5-43534cc5db99-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq\" (UID: \"03f63f69-2753-4d7d-9fd5-43534cc5db99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:37:54.362691 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.362658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fncrh\" (UniqueName: \"kubernetes.io/projected/03f63f69-2753-4d7d-9fd5-43534cc5db99-kube-api-access-fncrh\") pod \"maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq\" (UID: \"03f63f69-2753-4d7d-9fd5-43534cc5db99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:37:54.362755 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.362738    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f63f69-2753-4d7d-9fd5-43534cc5db99-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq\" (UID: \"03f63f69-2753-4d7d-9fd5-43534cc5db99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:37:54.417314 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.417274    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw" event={"ID":"7eb684ff-51f2-49c9-89e6-69fc847306cc","Type":"ContainerStarted","Data":"e4b0d3e2beb836de8d4985e7698ea47072ee1f99d1d42ff182c5dc71ec838032"}
Sep 09 15:37:54.418767 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.418746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7472-140647083561520" event={"ID":"02e015d4-754d-4009-91d4-ac26f8af2f4e","Type":"ContainerStarted","Data":"ba58c0ffe44b5fe7f11035300b5c290d317ec4d15e01adf7d984fe2f5bd07047"}
Sep 09 15:37:54.420045 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.420024    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41e5db374fbb968f7fd0980f0443abd88e07bb07434998c0af4722883fe968d8"
Sep 09 15:37:54.420120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.420077    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-140647083564080"
Sep 09 15:37:54.431467 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:37:54.431434    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7472-140647083564080\" is forbidden: User \"system:node:ip-10-0-134-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-183.ec2.internal' and this object" podUID="275ea03a-c215-441a-8a09-f23bdb9dee7a" pod="opendatahub/test-curl-7472-140647083564080"
Sep 09 15:37:54.463620 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.463590    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f63f69-2753-4d7d-9fd5-43534cc5db99-metrics-certs\") pod \"maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq\" (UID: \"03f63f69-2753-4d7d-9fd5-43534cc5db99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:37:54.464020 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.463642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f63f69-2753-4d7d-9fd5-43534cc5db99-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq\" (UID: \"03f63f69-2753-4d7d-9fd5-43534cc5db99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:37:54.464020 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.463675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fncrh\" (UniqueName: \"kubernetes.io/projected/03f63f69-2753-4d7d-9fd5-43534cc5db99-kube-api-access-fncrh\") pod \"maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq\" (UID: \"03f63f69-2753-4d7d-9fd5-43534cc5db99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:37:54.464020 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.463729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f63f69-2753-4d7d-9fd5-43534cc5db99-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq\" (UID: \"03f63f69-2753-4d7d-9fd5-43534cc5db99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:37:54.464020 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:37:54.463847    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-serving-cert: secret "maas-api-e2e-worker-w4-1e15c8-serving-cert" not found
Sep 09 15:37:54.464020 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:37:54.463910    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f63f69-2753-4d7d-9fd5-43534cc5db99-maas-api-tls podName:03f63f69-2753-4d7d-9fd5-43534cc5db99 nodeName:}" failed. No retries permitted until 2026-09-09 15:37:54.963892074 +0000 UTC m=+1064.605934054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03f63f69-2753-4d7d-9fd5-43534cc5db99-maas-api-tls") pod "maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq" (UID: "03f63f69-2753-4d7d-9fd5-43534cc5db99") : secret "maas-api-e2e-worker-w4-1e15c8-serving-cert" not found
Sep 09 15:37:54.464388 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.464371    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f63f69-2753-4d7d-9fd5-43534cc5db99-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq\" (UID: \"03f63f69-2753-4d7d-9fd5-43534cc5db99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:37:54.466180 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.466161    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f63f69-2753-4d7d-9fd5-43534cc5db99-metrics-certs\") pod \"maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq\" (UID: \"03f63f69-2753-4d7d-9fd5-43534cc5db99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:37:54.471840 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.471821    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fncrh\" (UniqueName: \"kubernetes.io/projected/03f63f69-2753-4d7d-9fd5-43534cc5db99-kube-api-access-fncrh\") pod \"maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq\" (UID: \"03f63f69-2753-4d7d-9fd5-43534cc5db99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:37:54.869259 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.869221    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="275ea03a-c215-441a-8a09-f23bdb9dee7a" path="/var/lib/kubelet/pods/275ea03a-c215-441a-8a09-f23bdb9dee7a/volumes"
Sep 09 15:37:54.968257 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.968218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f63f69-2753-4d7d-9fd5-43534cc5db99-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq\" (UID: \"03f63f69-2753-4d7d-9fd5-43534cc5db99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:37:54.971035 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:54.971009    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f63f69-2753-4d7d-9fd5-43534cc5db99-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq\" (UID: \"03f63f69-2753-4d7d-9fd5-43534cc5db99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:37:55.177503 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:55.177459    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:37:55.427241 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:55.427205    2578 generic.go:356] "Generic (PLEG): container finished" podID="02e015d4-754d-4009-91d4-ac26f8af2f4e" containerID="5396c3fb7f4af546c42403cc45afec3011b0468bb1a449cbe53239ab9a8bfd7d" exitCode=0
Sep 09 15:37:55.427409 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:55.427245    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7472-140647083561520" event={"ID":"02e015d4-754d-4009-91d4-ac26f8af2f4e","Type":"ContainerDied","Data":"5396c3fb7f4af546c42403cc45afec3011b0468bb1a449cbe53239ab9a8bfd7d"}
Sep 09 15:37:55.428936 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:55.428862    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw" event={"ID":"7eb684ff-51f2-49c9-89e6-69fc847306cc","Type":"ContainerStarted","Data":"a41ef767da8e8a944cb92398c5090fcced5516304a937b990aeeb1d4cf874f96"}
Sep 09 15:37:55.429090 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:55.429014    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:37:55.448347 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:55.448161    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"]
Sep 09 15:37:55.451232 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:37:55.451209    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f63f69_2753_4d7d_9fd5_43534cc5db99.slice/crio-a6e3e103baadabfab0ca8fc72df461ffc051bfc6c62525f0a1c2762540185773 WatchSource:0}: Error finding container a6e3e103baadabfab0ca8fc72df461ffc051bfc6c62525f0a1c2762540185773: Status 404 returned error can't find the container with id a6e3e103baadabfab0ca8fc72df461ffc051bfc6c62525f0a1c2762540185773
Sep 09 15:37:55.463477 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:55.463434    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw" podStartSLOduration=1.827442786 podStartE2EDuration="2.463419622s" totalImagesPullingTime="635.976836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:37:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:37:55.461500978 +0000 UTC m=+1065.103542967" watchObservedRunningTime="2026-09-09 15:37:55.463419622 +0000 UTC m=+1065.105461611"
Sep 09 15:37:56.436228 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:56.436188    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq" event={"ID":"03f63f69-2753-4d7d-9fd5-43534cc5db99","Type":"ContainerStarted","Data":"8a090b1646ad95220ce3dda3b0eb61406effc0682b8d0ef544fb20452a9b1832"}
Sep 09 15:37:56.436687 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:56.436282    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq" event={"ID":"03f63f69-2753-4d7d-9fd5-43534cc5db99","Type":"ContainerStarted","Data":"a6e3e103baadabfab0ca8fc72df461ffc051bfc6c62525f0a1c2762540185773"}
Sep 09 15:37:56.456378 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:56.456330    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq" podStartSLOduration=1.926544454 podStartE2EDuration="2.456315601s" totalImagesPullingTime="529.771147ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:37:56.456235664 +0000 UTC m=+1066.098277670" watchObservedRunningTime="2026-09-09 15:37:56.456315601 +0000 UTC m=+1066.098357601"
Sep 09 15:37:56.576290 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:56.576262    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-140647083561520"
Sep 09 15:37:56.607689 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:56.607649    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7472-140647083561520_02e015d4-754d-4009-91d4-ac26f8af2f4e/test-curl-7472-140647083561520/0.log"
Sep 09 15:37:56.635420 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:56.635390    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7472-140647083561520"]
Sep 09 15:37:56.639333 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:56.639305    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7472-140647083561520"]
Sep 09 15:37:56.686646 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:56.686570    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnh7q\" (UniqueName: \"kubernetes.io/projected/02e015d4-754d-4009-91d4-ac26f8af2f4e-kube-api-access-xnh7q\") pod \"02e015d4-754d-4009-91d4-ac26f8af2f4e\" (UID: \"02e015d4-754d-4009-91d4-ac26f8af2f4e\") "
Sep 09 15:37:56.688780 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:56.688755    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e015d4-754d-4009-91d4-ac26f8af2f4e-kube-api-access-xnh7q" (OuterVolumeSpecName: "kube-api-access-xnh7q") pod "02e015d4-754d-4009-91d4-ac26f8af2f4e" (UID: "02e015d4-754d-4009-91d4-ac26f8af2f4e"). InnerVolumeSpecName "kube-api-access-xnh7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:37:56.788288 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:56.788254    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnh7q\" (UniqueName: \"kubernetes.io/projected/02e015d4-754d-4009-91d4-ac26f8af2f4e-kube-api-access-xnh7q\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:37:56.835880 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:56.835845    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:56.867553 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:56.867522    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02e015d4-754d-4009-91d4-ac26f8af2f4e" path="/var/lib/kubelet/pods/02e015d4-754d-4009-91d4-ac26f8af2f4e/volumes"
Sep 09 15:37:56.937524 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:56.937448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:57.037460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.037424    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:57.207272 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.207192    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:57.306748 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.306717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:57.444102 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.444066    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-140647083561520"
Sep 09 15:37:57.444562 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.444109    2578 scope.go:117] "RemoveContainer" containerID="5396c3fb7f4af546c42403cc45afec3011b0468bb1a449cbe53239ab9a8bfd7d"
Sep 09 15:37:57.444562 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.444448    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:37:57.461823 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.461763    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:57.547821 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.547790    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"]
Sep 09 15:37:57.548257 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.548244    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e015d4-754d-4009-91d4-ac26f8af2f4e" containerName="test-curl-7472-140647083561520"
Sep 09 15:37:57.548316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.548260    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e015d4-754d-4009-91d4-ac26f8af2f4e" containerName="test-curl-7472-140647083561520"
Sep 09 15:37:57.548352 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.548322    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e015d4-754d-4009-91d4-ac26f8af2f4e" containerName="test-curl-7472-140647083561520"
Sep 09 15:37:57.553129 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.553104    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.555570 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.555546    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-8a43b2-openshift-default-dockercfg-gr8ds\""
Sep 09 15:37:57.564312 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.564291    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"]
Sep 09 15:37:57.580664 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.580633    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:57.697557 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.697521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z74z9\" (UniqueName: \"kubernetes.io/projected/0e9550f7-8239-4de0-afe3-c793a9f7fa56-kube-api-access-z74z9\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.697729 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.697608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-workload-certs\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.697729 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.697658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-podinfo\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.697729 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.697690    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-envoy\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.697729 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.697721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-token\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.697865 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.697759    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istiod-ca-cert\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.697865 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.697815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-workload-socket\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.697865 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.697850    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-data\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.697958 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.697922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-credential-socket\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.798781 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.798706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-workload-certs\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.798781 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.798748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-podinfo\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.798781 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.798766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-envoy\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.798781 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.798783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-token\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.799129 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.798801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istiod-ca-cert\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.799129 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.798833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-workload-socket\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.799129 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.798852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-data\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.799129 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.798935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-credential-socket\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.799129 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.798972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z74z9\" (UniqueName: \"kubernetes.io/projected/0e9550f7-8239-4de0-afe3-c793a9f7fa56-kube-api-access-z74z9\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.799380 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.799193    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-workload-certs\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.799380 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.799342    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-workload-socket\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.799635 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.799381    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-credential-socket\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.799635 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.799420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-data\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.799635 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.799563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istiod-ca-cert\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.801224 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.801199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-envoy\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.801531 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.801510    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-podinfo\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.811546 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.811519    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z74z9\" (UniqueName: \"kubernetes.io/projected/0e9550f7-8239-4de0-afe3-c793a9f7fa56-kube-api-access-z74z9\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.818241 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.818220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-token\") pod \"e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") " pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.864220 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.864190    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:57.875873 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.875847    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:57.972323 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:57.972279    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:37:58.158456 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:58.158429    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"]
Sep 09 15:37:58.160430 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:37:58.160400    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9550f7_8239_4de0_afe3_c793a9f7fa56.slice/crio-2f4e9ed5eeff4843c27fa49814b3c7bf2e45b7c637cab8977f0b0296fa8071bf WatchSource:0}: Error finding container 2f4e9ed5eeff4843c27fa49814b3c7bf2e45b7c637cab8977f0b0296fa8071bf: Status 404 returned error can't find the container with id 2f4e9ed5eeff4843c27fa49814b3c7bf2e45b7c637cab8977f0b0296fa8071bf
Sep 09 15:37:58.162671 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:58.162640    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 15:37:58.162760 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:58.162720    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 15:37:58.162873 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:58.162765    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 15:37:58.451885 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:58.451799    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq" event={"ID":"0e9550f7-8239-4de0-afe3-c793a9f7fa56","Type":"ContainerStarted","Data":"bab3f494e98181d551701c8e54933438b0d051eb6e91eefc24fb7db91e7531ac"}
Sep 09 15:37:58.451885 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:58.451857    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq" event={"ID":"0e9550f7-8239-4de0-afe3-c793a9f7fa56","Type":"ContainerStarted","Data":"2f4e9ed5eeff4843c27fa49814b3c7bf2e45b7c637cab8977f0b0296fa8071bf"}
Sep 09 15:37:58.473379 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:58.473328    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq" podStartSLOduration=1.47331294 podStartE2EDuration="1.47331294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:37:58.471309503 +0000 UTC m=+1068.113351495" watchObservedRunningTime="2026-09-09 15:37:58.47331294 +0000 UTC m=+1068.115354930"
Sep 09 15:37:58.873678 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:58.873600    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:37:59.870295 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:37:59.870267    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:38:00.464061 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:00.464028    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:38:00.465019 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:00.464995    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:38:00.534743 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:00.534641    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:00.627631 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:00.627593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:01.113599 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.113561    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:01.209858 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.209829    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:01.352797 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.352760    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:01.444561 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.444531    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:38:01.651964 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.651885    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"]
Sep 09 15:38:01.656116 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.656072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:38:01.658417 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.658396    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-8a43b2-serving-cert\""
Sep 09 15:38:01.665495 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.665468    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"]
Sep 09 15:38:01.688407 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.688380    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:01.737912 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.737830    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-openshift-service-ca\") pod \"maas-api-e2e-autores-8a43b2-7dd8464c4-d686d\" (UID: \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:38:01.737912 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.737865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52fkg\" (UniqueName: \"kubernetes.io/projected/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-kube-api-access-52fkg\") pod \"maas-api-e2e-autores-8a43b2-7dd8464c4-d686d\" (UID: \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:38:01.738162 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.737965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-metrics-certs\") pod \"maas-api-e2e-autores-8a43b2-7dd8464c4-d686d\" (UID: \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:38:01.738162 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.738064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-maas-api-tls\") pod \"maas-api-e2e-autores-8a43b2-7dd8464c4-d686d\" (UID: \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:38:01.789132 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.789101    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:01.839259 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.839224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-openshift-service-ca\") pod \"maas-api-e2e-autores-8a43b2-7dd8464c4-d686d\" (UID: \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:38:01.839259 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.839262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52fkg\" (UniqueName: \"kubernetes.io/projected/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-kube-api-access-52fkg\") pod \"maas-api-e2e-autores-8a43b2-7dd8464c4-d686d\" (UID: \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:38:01.839497 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.839329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-metrics-certs\") pod \"maas-api-e2e-autores-8a43b2-7dd8464c4-d686d\" (UID: \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:38:01.839497 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.839367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-maas-api-tls\") pod \"maas-api-e2e-autores-8a43b2-7dd8464c4-d686d\" (UID: \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:38:01.839497 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:38:01.839484    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-serving-cert: secret "maas-api-e2e-autores-8a43b2-serving-cert" not found
Sep 09 15:38:01.839726 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:38:01.839536    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-maas-api-tls podName:c68e5346-e4e9-4e09-af2f-98cb26f3cce7 nodeName:}" failed. No retries permitted until 2026-09-09 15:38:02.339517211 +0000 UTC m=+1071.981559179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-maas-api-tls") pod "maas-api-e2e-autores-8a43b2-7dd8464c4-d686d" (UID: "c68e5346-e4e9-4e09-af2f-98cb26f3cce7") : secret "maas-api-e2e-autores-8a43b2-serving-cert" not found
Sep 09 15:38:01.839894 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.839876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-openshift-service-ca\") pod \"maas-api-e2e-autores-8a43b2-7dd8464c4-d686d\" (UID: \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:38:01.841903 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.841879    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-metrics-certs\") pod \"maas-api-e2e-autores-8a43b2-7dd8464c4-d686d\" (UID: \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:38:01.854193 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.854171    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52fkg\" (UniqueName: \"kubernetes.io/projected/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-kube-api-access-52fkg\") pod \"maas-api-e2e-autores-8a43b2-7dd8464c4-d686d\" (UID: \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:38:01.946215 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:01.946185    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:02.312003 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:02.311432    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:02.346005 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:02.345951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-maas-api-tls\") pod \"maas-api-e2e-autores-8a43b2-7dd8464c4-d686d\" (UID: \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:38:02.348936 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:02.348907    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-maas-api-tls\") pod \"maas-api-e2e-autores-8a43b2-7dd8464c4-d686d\" (UID: \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:38:02.409004 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:02.408355    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:02.571028 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:02.570906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:38:02.874180 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:38:02.874149    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc68e5346_e4e9_4e09_af2f_98cb26f3cce7.slice/crio-40219e959635c3776933f6c9b341b0c625e4e19098c9037d627e2065ffeb5d4f WatchSource:0}: Error finding container 40219e959635c3776933f6c9b341b0c625e4e19098c9037d627e2065ffeb5d4f: Status 404 returned error can't find the container with id 40219e959635c3776933f6c9b341b0c625e4e19098c9037d627e2065ffeb5d4f
Sep 09 15:38:02.876428 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:02.876402    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"]
Sep 09 15:38:03.459586 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:03.459564    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:38:03.483716 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:03.483682    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d" event={"ID":"c68e5346-e4e9-4e09-af2f-98cb26f3cce7","Type":"ContainerStarted","Data":"150d0d22e8cbc48a3996787fbd6bbf180441414d0319a2135bb6bc54a9fa9b6b"}
Sep 09 15:38:03.483890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:03.483739    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d" event={"ID":"c68e5346-e4e9-4e09-af2f-98cb26f3cce7","Type":"ContainerStarted","Data":"40219e959635c3776933f6c9b341b0c625e4e19098c9037d627e2065ffeb5d4f"}
Sep 09 15:38:03.483937 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:03.483887    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:38:03.517767 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:03.502693    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d" podStartSLOduration=2.01259615 podStartE2EDuration="2.502670918s" totalImagesPullingTime="490.074768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:38:03.499108945 +0000 UTC m=+1073.141150936" watchObservedRunningTime="2026-09-09 15:38:03.502670918 +0000 UTC m=+1073.144712909"
Sep 09 15:38:03.956149 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:03.956110    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:04.058530 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:04.058489    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:05.108626 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.108585    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"]
Sep 09 15:38:05.113818 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.113792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"
Sep 09 15:38:05.116443 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.116317    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1c76a304-serving-cert\""
Sep 09 15:38:05.129099 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.128653    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"]
Sep 09 15:38:05.174476 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.174443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx6m2\" (UniqueName: \"kubernetes.io/projected/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-kube-api-access-cx6m2\") pod \"maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q\" (UID: \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"
Sep 09 15:38:05.174629 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.174482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-metrics-certs\") pod \"maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q\" (UID: \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"
Sep 09 15:38:05.174629 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.174536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-maas-api-tls\") pod \"maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q\" (UID: \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"
Sep 09 15:38:05.174700 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.174638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-openshift-service-ca\") pod \"maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q\" (UID: \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"
Sep 09 15:38:05.213048 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.212968    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:05.275331 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.275295    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx6m2\" (UniqueName: \"kubernetes.io/projected/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-kube-api-access-cx6m2\") pod \"maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q\" (UID: \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"
Sep 09 15:38:05.275331 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.275334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-metrics-certs\") pod \"maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q\" (UID: \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"
Sep 09 15:38:05.275587 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.275375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-maas-api-tls\") pod \"maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q\" (UID: \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"
Sep 09 15:38:05.275587 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.275448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-openshift-service-ca\") pod \"maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q\" (UID: \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"
Sep 09 15:38:05.276177 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.276143    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-openshift-service-ca\") pod \"maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q\" (UID: \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"
Sep 09 15:38:05.277949 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.277928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-metrics-certs\") pod \"maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q\" (UID: \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"
Sep 09 15:38:05.278700 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.278681    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-maas-api-tls\") pod \"maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q\" (UID: \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"
Sep 09 15:38:05.284833 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.284810    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx6m2\" (UniqueName: \"kubernetes.io/projected/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-kube-api-access-cx6m2\") pod \"maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q\" (UID: \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"
Sep 09 15:38:05.295951 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.295893    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:05.430580 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.430542    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"
Sep 09 15:38:05.715562 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.715408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:05.726059 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.726035    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"]
Sep 09 15:38:05.728197 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:38:05.728169    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73d2be8_cfc6_4c0f_8552_476bd9bc9534.slice/crio-47247c54b00bcc0e30c704687c105979f03cf19d180a5393bbb01b9a11b45a80 WatchSource:0}: Error finding container 47247c54b00bcc0e30c704687c105979f03cf19d180a5393bbb01b9a11b45a80: Status 404 returned error can't find the container with id 47247c54b00bcc0e30c704687c105979f03cf19d180a5393bbb01b9a11b45a80
Sep 09 15:38:05.814001 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:05.813954    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:06.534186 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:06.534148    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q" event={"ID":"a73d2be8-cfc6-4c0f-8552-476bd9bc9534","Type":"ContainerStarted","Data":"6eab84f42a3b193aa6bb10b5146281bc5d17707e874d850913589fee102da668"}
Sep 09 15:38:06.534548 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:06.534205    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q" event={"ID":"a73d2be8-cfc6-4c0f-8552-476bd9bc9534","Type":"ContainerStarted","Data":"47247c54b00bcc0e30c704687c105979f03cf19d180a5393bbb01b9a11b45a80"}
Sep 09 15:38:06.534548 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:06.534356    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"
Sep 09 15:38:06.552493 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:06.552437    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q" podStartSLOduration=0.935602946 podStartE2EDuration="1.552422304s" totalImagesPullingTime="616.819358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:38:06.55105486 +0000 UTC m=+1076.193096849" watchObservedRunningTime="2026-09-09 15:38:06.552422304 +0000 UTC m=+1076.194464303"
Sep 09 15:38:07.295520 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:07.295466    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:07.520490 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:07.520446    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:07.608108 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:07.608006    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:07.708544 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:07.708517    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:07.995665 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:07.995632    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:08.076770 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:08.076721    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:08.237592 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:08.237558    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:08.469534 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:08.469458    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:08.604627 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:08.604595    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:09.497165 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:09.497137    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:38:11.569253 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:11.569217    2578 generic.go:356] "Generic (PLEG): container finished" podID="5bb883ca-8542-4127-b9be-2ba9571c8c93" containerID="07613ebb5db0edf16932f290c59c23c0630f9c62e3a87b551fb0616e5f68637c" exitCode=0
Sep 09 15:38:11.569684 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:11.569294    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk" event={"ID":"5bb883ca-8542-4127-b9be-2ba9571c8c93","Type":"ContainerDied","Data":"07613ebb5db0edf16932f290c59c23c0630f9c62e3a87b551fb0616e5f68637c"}
Sep 09 15:38:11.569756 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:11.569743    2578 scope.go:117] "RemoveContainer" containerID="07613ebb5db0edf16932f290c59c23c0630f9c62e3a87b551fb0616e5f68637c"
Sep 09 15:38:11.570865 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:11.570838    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 15:38:11.571071 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:11.571053    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 15:38:11.571119 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:11.571094    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 15:38:12.547780 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:12.547753    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"
Sep 09 15:38:12.579805 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:12.579766    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk" event={"ID":"5bb883ca-8542-4127-b9be-2ba9571c8c93","Type":"ContainerStarted","Data":"a851a9861ce1ec900611ac7ececfed46cf706103849e8fb8666957ec3ea69eba"}
Sep 09 15:38:13.050787 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:13.050756    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:38:13.051083 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:13.051061    2578 patch_prober.go:28] interesting pod/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 09 15:38:13.051192 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:13.051111    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk" podUID="5bb883ca-8542-4127-b9be-2ba9571c8c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 09 15:38:14.050993 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:14.050940    2578 patch_prober.go:28] interesting pod/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 09 15:38:14.051404 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:14.051034    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk" podUID="5bb883ca-8542-4127-b9be-2ba9571c8c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 09 15:38:14.226843 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:14.226794    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:14.328138 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:14.328039    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:14.427264 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:14.427214    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:15.050339 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:15.050304    2578 patch_prober.go:28] interesting pod/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 09 15:38:15.050549 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:15.050367    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk" podUID="5bb883ca-8542-4127-b9be-2ba9571c8c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 09 15:38:15.161739 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:15.161709    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:15.248871 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:15.248817    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:15.356145 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:15.356050    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:16.050456 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:16.050428    2578 patch_prober.go:28] interesting pod/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 09 15:38:16.050662 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:16.050630    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk" podUID="5bb883ca-8542-4127-b9be-2ba9571c8c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 09 15:38:16.219675 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:16.219612    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:17.050563 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:17.050526    2578 patch_prober.go:28] interesting pod/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 09 15:38:17.050747 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:17.050581    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk" podUID="5bb883ca-8542-4127-b9be-2ba9571c8c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 09 15:38:17.135952 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:17.135918    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:17.236543 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:17.236474    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:17.345049 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:17.344920    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:17.753371 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:17.753337    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:17.840690 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:17.840659    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:18.050372 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:18.050272    2578 patch_prober.go:28] interesting pod/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 09 15:38:18.050372 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:18.050355    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk" podUID="5bb883ca-8542-4127-b9be-2ba9571c8c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 09 15:38:18.732612 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:18.732574    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:19.054609 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:19.054531    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:38:19.054942 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:19.054923    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:38:19.055637 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:19.055619    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:38:19.786591 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:19.786564    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:22.717438 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:22.717412    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:22.818235 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:22.818202    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:23.222731 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:23.222694    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:23.654910 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:23.654872    2578 generic.go:356] "Generic (PLEG): container finished" podID="0e9550f7-8239-4de0-afe3-c793a9f7fa56" containerID="bab3f494e98181d551701c8e54933438b0d051eb6e91eefc24fb7db91e7531ac" exitCode=0
Sep 09 15:38:23.655095 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:23.654950    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq" event={"ID":"0e9550f7-8239-4de0-afe3-c793a9f7fa56","Type":"ContainerDied","Data":"bab3f494e98181d551701c8e54933438b0d051eb6e91eefc24fb7db91e7531ac"}
Sep 09 15:38:23.655383 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:23.655364    2578 scope.go:117] "RemoveContainer" containerID="bab3f494e98181d551701c8e54933438b0d051eb6e91eefc24fb7db91e7531ac"
Sep 09 15:38:23.656495 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:23.656471    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 15:38:23.656562 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:23.656533    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 15:38:23.656597 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:23.656561    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 15:38:23.746322 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:23.746288    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:24.142486 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:24.142445    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:24.662459 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:24.662425    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq" event={"ID":"0e9550f7-8239-4de0-afe3-c793a9f7fa56","Type":"ContainerStarted","Data":"b78ce0c46d1dec1da891d43ffd503572b3f1bffe59780c1329f0789a68b99d20"}
Sep 09 15:38:24.867925 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:24.867886    2578 patch_prober.go:28] interesting pod/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 09 15:38:24.868372 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:24.868004    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq" podUID="0e9550f7-8239-4de0-afe3-c793a9f7fa56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 09 15:38:24.870857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:24.870835    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:38:25.580964 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:25.580930    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:25.865092 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:25.864974    2578 patch_prober.go:28] interesting pod/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 09 15:38:25.865092 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:25.865060    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq" podUID="0e9550f7-8239-4de0-afe3-c793a9f7fa56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 09 15:38:26.864603 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:26.864567    2578 patch_prober.go:28] interesting pod/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 09 15:38:26.865170 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:26.865139    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq" podUID="0e9550f7-8239-4de0-afe3-c793a9f7fa56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 09 15:38:27.864494 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:27.864458    2578 patch_prober.go:28] interesting pod/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 09 15:38:27.864668 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:27.864522    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq" podUID="0e9550f7-8239-4de0-afe3-c793a9f7fa56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 09 15:38:28.814672 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:28.814637    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"]
Sep 09 15:38:28.820207 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:28.820178    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:38:28.822359 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:28.822337    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-267962-serving-cert\""
Sep 09 15:38:28.829335 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:28.829310    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"]
Sep 09 15:38:28.845925 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:28.845901    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b70461-1b17-4739-8660-51f9b4f2a2d9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-267962-58d68778cf-p5x4p\" (UID: \"67b70461-1b17-4739-8660-51f9b4f2a2d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:38:28.846090 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:28.845997    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4nls\" (UniqueName: \"kubernetes.io/projected/67b70461-1b17-4739-8660-51f9b4f2a2d9-kube-api-access-d4nls\") pod \"maas-api-e2e-inf-b-267962-58d68778cf-p5x4p\" (UID: \"67b70461-1b17-4739-8660-51f9b4f2a2d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:38:28.846150 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:28.846116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67b70461-1b17-4739-8660-51f9b4f2a2d9-maas-api-tls\") pod \"maas-api-e2e-inf-b-267962-58d68778cf-p5x4p\" (UID: \"67b70461-1b17-4739-8660-51f9b4f2a2d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:38:28.846256 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:28.846238    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b70461-1b17-4739-8660-51f9b4f2a2d9-metrics-certs\") pod \"maas-api-e2e-inf-b-267962-58d68778cf-p5x4p\" (UID: \"67b70461-1b17-4739-8660-51f9b4f2a2d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:38:28.865095 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:28.865066    2578 patch_prober.go:28] interesting pod/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 09 15:38:28.865436 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:28.865124    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq" podUID="0e9550f7-8239-4de0-afe3-c793a9f7fa56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 09 15:38:28.947705 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:28.947670    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b70461-1b17-4739-8660-51f9b4f2a2d9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-267962-58d68778cf-p5x4p\" (UID: \"67b70461-1b17-4739-8660-51f9b4f2a2d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:38:28.947705 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:28.947706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4nls\" (UniqueName: \"kubernetes.io/projected/67b70461-1b17-4739-8660-51f9b4f2a2d9-kube-api-access-d4nls\") pod \"maas-api-e2e-inf-b-267962-58d68778cf-p5x4p\" (UID: \"67b70461-1b17-4739-8660-51f9b4f2a2d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:38:28.947960 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:28.947777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67b70461-1b17-4739-8660-51f9b4f2a2d9-maas-api-tls\") pod \"maas-api-e2e-inf-b-267962-58d68778cf-p5x4p\" (UID: \"67b70461-1b17-4739-8660-51f9b4f2a2d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:38:28.947960 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:28.947840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b70461-1b17-4739-8660-51f9b4f2a2d9-metrics-certs\") pod \"maas-api-e2e-inf-b-267962-58d68778cf-p5x4p\" (UID: \"67b70461-1b17-4739-8660-51f9b4f2a2d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:38:28.948220 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:38:28.948196    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-serving-cert: secret "maas-api-e2e-inf-b-267962-serving-cert" not found
Sep 09 15:38:28.948294 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:38:28.948271    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67b70461-1b17-4739-8660-51f9b4f2a2d9-maas-api-tls podName:67b70461-1b17-4739-8660-51f9b4f2a2d9 nodeName:}" failed. No retries permitted until 2026-09-09 15:38:29.448253036 +0000 UTC m=+1099.090295007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/67b70461-1b17-4739-8660-51f9b4f2a2d9-maas-api-tls") pod "maas-api-e2e-inf-b-267962-58d68778cf-p5x4p" (UID: "67b70461-1b17-4739-8660-51f9b4f2a2d9") : secret "maas-api-e2e-inf-b-267962-serving-cert" not found
Sep 09 15:38:28.948365 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:28.948312    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b70461-1b17-4739-8660-51f9b4f2a2d9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-267962-58d68778cf-p5x4p\" (UID: \"67b70461-1b17-4739-8660-51f9b4f2a2d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:38:28.951016 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:28.950952    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b70461-1b17-4739-8660-51f9b4f2a2d9-metrics-certs\") pod \"maas-api-e2e-inf-b-267962-58d68778cf-p5x4p\" (UID: \"67b70461-1b17-4739-8660-51f9b4f2a2d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:38:28.961450 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:28.961424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4nls\" (UniqueName: \"kubernetes.io/projected/67b70461-1b17-4739-8660-51f9b4f2a2d9-kube-api-access-d4nls\") pod \"maas-api-e2e-inf-b-267962-58d68778cf-p5x4p\" (UID: \"67b70461-1b17-4739-8660-51f9b4f2a2d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:38:29.152633 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:29.152589    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:29.253182 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:29.253139    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:29.364125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:29.364099    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:29.453929 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:29.453845    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67b70461-1b17-4739-8660-51f9b4f2a2d9-maas-api-tls\") pod \"maas-api-e2e-inf-b-267962-58d68778cf-p5x4p\" (UID: \"67b70461-1b17-4739-8660-51f9b4f2a2d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:38:29.456209 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:29.456184    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67b70461-1b17-4739-8660-51f9b4f2a2d9-maas-api-tls\") pod \"maas-api-e2e-inf-b-267962-58d68778cf-p5x4p\" (UID: \"67b70461-1b17-4739-8660-51f9b4f2a2d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:38:29.460465 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:29.460395    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:29.737194 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:29.737108    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:38:29.869323 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:29.869292    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:38:29.869754 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:29.869557    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:38:29.870385 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:29.870367    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:38:29.985850 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:29.985820    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:30.011066 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:30.011045    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"]
Sep 09 15:38:30.014018 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:38:30.013962    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b70461_1b17_4739_8660_51f9b4f2a2d9.slice/crio-5120aba5f23e2427eb48e15f2e03952f3da473bbd759f70df7e7faab043a3061 WatchSource:0}: Error finding container 5120aba5f23e2427eb48e15f2e03952f3da473bbd759f70df7e7faab043a3061: Status 404 returned error can't find the container with id 5120aba5f23e2427eb48e15f2e03952f3da473bbd759f70df7e7faab043a3061
Sep 09 15:38:30.699417 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:30.699388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p" event={"ID":"67b70461-1b17-4739-8660-51f9b4f2a2d9","Type":"ContainerStarted","Data":"5120aba5f23e2427eb48e15f2e03952f3da473bbd759f70df7e7faab043a3061"}
Sep 09 15:38:30.763495 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:30.763461    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:33.716868 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:33.716769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p" event={"ID":"67b70461-1b17-4739-8660-51f9b4f2a2d9","Type":"ContainerStarted","Data":"e43a351f12f37f16182b7ee4b3015f06a0f9a91435a09b3f726d0c71d2d20365"}
Sep 09 15:38:33.717321 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:33.716930    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:38:33.732671 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:33.732623    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p" podStartSLOduration=2.358738994 podStartE2EDuration="5.73260766s" totalImagesPullingTime="3.373868666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:38:33.731372014 +0000 UTC m=+1103.373414023" watchObservedRunningTime="2026-09-09 15:38:33.73260766 +0000 UTC m=+1103.374649648"
Sep 09 15:38:35.740019 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.739967    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"]
Sep 09 15:38:35.743746 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.743726    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:35.746878 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.746856    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-df30be\"/\"default-dockercfg-gdgnh\""
Sep 09 15:38:35.747004 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.746901    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-df30be\"/\"simulator-w0-df30be-kserve-self-signed-certs\""
Sep 09 15:38:35.747004 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.746855    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-df30be\"/\"openshift-service-ca.crt\""
Sep 09 15:38:35.747004 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.746994    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-df30be\"/\"kube-root-ca.crt\""
Sep 09 15:38:35.750868 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.750846    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"]
Sep 09 15:38:35.817647 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.817618    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-dshm\") pod \"simulator-w0-df30be-kserve-cdc86ff49-7zb4h\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") " pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:35.817819 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.817680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-home\") pod \"simulator-w0-df30be-kserve-cdc86ff49-7zb4h\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") " pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:35.817819 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.817695    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-model-cache\") pod \"simulator-w0-df30be-kserve-cdc86ff49-7zb4h\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") " pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:35.817819 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.817722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n42c2\" (UniqueName: \"kubernetes.io/projected/fbf28447-9239-4a6e-adfe-7baa6e38ece4-kube-api-access-n42c2\") pod \"simulator-w0-df30be-kserve-cdc86ff49-7zb4h\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") " pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:35.817819 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.817793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf28447-9239-4a6e-adfe-7baa6e38ece4-tls-certs\") pod \"simulator-w0-df30be-kserve-cdc86ff49-7zb4h\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") " pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:35.817967 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.817855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-tmp-dir\") pod \"simulator-w0-df30be-kserve-cdc86ff49-7zb4h\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") " pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:35.918625 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.918588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf28447-9239-4a6e-adfe-7baa6e38ece4-tls-certs\") pod \"simulator-w0-df30be-kserve-cdc86ff49-7zb4h\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") " pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:35.918814 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.918689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-tmp-dir\") pod \"simulator-w0-df30be-kserve-cdc86ff49-7zb4h\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") " pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:35.918814 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.918722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-dshm\") pod \"simulator-w0-df30be-kserve-cdc86ff49-7zb4h\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") " pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:35.918814 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.918804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-home\") pod \"simulator-w0-df30be-kserve-cdc86ff49-7zb4h\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") " pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:35.919021 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.918827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-model-cache\") pod \"simulator-w0-df30be-kserve-cdc86ff49-7zb4h\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") " pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:35.919021 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.918871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n42c2\" (UniqueName: \"kubernetes.io/projected/fbf28447-9239-4a6e-adfe-7baa6e38ece4-kube-api-access-n42c2\") pod \"simulator-w0-df30be-kserve-cdc86ff49-7zb4h\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") " pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:35.919519 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.919487    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-home\") pod \"simulator-w0-df30be-kserve-cdc86ff49-7zb4h\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") " pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:35.919672 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.919552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-model-cache\") pod \"simulator-w0-df30be-kserve-cdc86ff49-7zb4h\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") " pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:35.919850 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.919824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-tmp-dir\") pod \"simulator-w0-df30be-kserve-cdc86ff49-7zb4h\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") " pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:35.921446 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.921425    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-dshm\") pod \"simulator-w0-df30be-kserve-cdc86ff49-7zb4h\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") " pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:35.921674 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.921654    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf28447-9239-4a6e-adfe-7baa6e38ece4-tls-certs\") pod \"simulator-w0-df30be-kserve-cdc86ff49-7zb4h\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") " pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:35.926610 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:35.926586    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n42c2\" (UniqueName: \"kubernetes.io/projected/fbf28447-9239-4a6e-adfe-7baa6e38ece4-kube-api-access-n42c2\") pod \"simulator-w0-df30be-kserve-cdc86ff49-7zb4h\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") " pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:36.065081 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:36.064957    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:36.336453 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:36.336428    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"]
Sep 09 15:38:36.338344 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:38:36.338321    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbf28447_9239_4a6e_adfe_7baa6e38ece4.slice/crio-0512da21cfeeddf99ac569dbe67c78ec22e359de20658c79fbd74dfa7e7a412c WatchSource:0}: Error finding container 0512da21cfeeddf99ac569dbe67c78ec22e359de20658c79fbd74dfa7e7a412c: Status 404 returned error can't find the container with id 0512da21cfeeddf99ac569dbe67c78ec22e359de20658c79fbd74dfa7e7a412c
Sep 09 15:38:36.734197 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:36.734157    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h" event={"ID":"fbf28447-9239-4a6e-adfe-7baa6e38ece4","Type":"ContainerStarted","Data":"2a8a7bc158d79110cfe838879f1a542c9f38e30dd935fbd22238a1d741028cdd"}
Sep 09 15:38:36.734382 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:36.734213    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h" event={"ID":"fbf28447-9239-4a6e-adfe-7baa6e38ece4","Type":"ContainerStarted","Data":"0512da21cfeeddf99ac569dbe67c78ec22e359de20658c79fbd74dfa7e7a412c"}
Sep 09 15:38:36.752370 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:36.752324    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h" podStartSLOduration=1.7523083210000001 podStartE2EDuration="1.752308321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:38:36.751346134 +0000 UTC m=+1106.393388139" watchObservedRunningTime="2026-09-09 15:38:36.752308321 +0000 UTC m=+1106.394350310"
Sep 09 15:38:38.874726 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:38.874699    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:39.727959 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:39.727927    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:38:40.335505 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:40.335468    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:45.578717 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:45.578683    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:46.065425 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.065389    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:46.065613 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.065437    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:46.078114 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.078085    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:46.320514 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.320442    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"]
Sep 09 15:38:46.320673 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.320653    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q" podUID="a73d2be8-cfc6-4c0f-8552-476bd9bc9534" containerName="maas-api" containerID="cri-o://6eab84f42a3b193aa6bb10b5146281bc5d17707e874d850913589fee102da668" gracePeriod=30
Sep 09 15:38:46.352848 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.352822    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:46.431772 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.431742    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:46.496736 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.496698    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:46.571895 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.571834    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"
Sep 09 15:38:46.628161 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.628131    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-metrics-certs\") pod \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\" (UID: \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\") "
Sep 09 15:38:46.628587 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.628288    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-openshift-service-ca\") pod \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\" (UID: \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\") "
Sep 09 15:38:46.628587 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.628350    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-maas-api-tls\") pod \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\" (UID: \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\") "
Sep 09 15:38:46.628587 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.628381    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cx6m2\" (UniqueName: \"kubernetes.io/projected/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-kube-api-access-cx6m2\") pod \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\" (UID: \"a73d2be8-cfc6-4c0f-8552-476bd9bc9534\") "
Sep 09 15:38:46.628772 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.628699    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a73d2be8-cfc6-4c0f-8552-476bd9bc9534" (UID: "a73d2be8-cfc6-4c0f-8552-476bd9bc9534"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:38:46.630791 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.630758    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a73d2be8-cfc6-4c0f-8552-476bd9bc9534" (UID: "a73d2be8-cfc6-4c0f-8552-476bd9bc9534"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:38:46.630791 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.630780    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-kube-api-access-cx6m2" (OuterVolumeSpecName: "kube-api-access-cx6m2") pod "a73d2be8-cfc6-4c0f-8552-476bd9bc9534" (UID: "a73d2be8-cfc6-4c0f-8552-476bd9bc9534"). InnerVolumeSpecName "kube-api-access-cx6m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:38:46.630969 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.630825    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a73d2be8-cfc6-4c0f-8552-476bd9bc9534" (UID: "a73d2be8-cfc6-4c0f-8552-476bd9bc9534"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:38:46.729733 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.729705    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-openshift-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:38:46.729733 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.729732    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-maas-api-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:38:46.729733 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.729741    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx6m2\" (UniqueName: \"kubernetes.io/projected/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-kube-api-access-cx6m2\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:38:46.729950 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.729749    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a73d2be8-cfc6-4c0f-8552-476bd9bc9534-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:38:46.772442 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.772415    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:46.787542 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.787512    2578 generic.go:356] "Generic (PLEG): container finished" podID="a73d2be8-cfc6-4c0f-8552-476bd9bc9534" containerID="6eab84f42a3b193aa6bb10b5146281bc5d17707e874d850913589fee102da668" exitCode=0
Sep 09 15:38:46.787692 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.787556    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"
Sep 09 15:38:46.787692 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.787587    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q" event={"ID":"a73d2be8-cfc6-4c0f-8552-476bd9bc9534","Type":"ContainerDied","Data":"6eab84f42a3b193aa6bb10b5146281bc5d17707e874d850913589fee102da668"}
Sep 09 15:38:46.787692 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.787628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q" event={"ID":"a73d2be8-cfc6-4c0f-8552-476bd9bc9534","Type":"ContainerDied","Data":"47247c54b00bcc0e30c704687c105979f03cf19d180a5393bbb01b9a11b45a80"}
Sep 09 15:38:46.787692 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.787648    2578 scope.go:117] "RemoveContainer" containerID="6eab84f42a3b193aa6bb10b5146281bc5d17707e874d850913589fee102da668"
Sep 09 15:38:46.797410 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.797384    2578 scope.go:117] "RemoveContainer" containerID="6eab84f42a3b193aa6bb10b5146281bc5d17707e874d850913589fee102da668"
Sep 09 15:38:46.797710 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:38:46.797689    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eab84f42a3b193aa6bb10b5146281bc5d17707e874d850913589fee102da668\": container with ID starting with 6eab84f42a3b193aa6bb10b5146281bc5d17707e874d850913589fee102da668 not found: ID does not exist" containerID="6eab84f42a3b193aa6bb10b5146281bc5d17707e874d850913589fee102da668"
Sep 09 15:38:46.797803 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.797716    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eab84f42a3b193aa6bb10b5146281bc5d17707e874d850913589fee102da668"} err="failed to get container status \"6eab84f42a3b193aa6bb10b5146281bc5d17707e874d850913589fee102da668\": rpc error: code = NotFound desc = could not find container \"6eab84f42a3b193aa6bb10b5146281bc5d17707e874d850913589fee102da668\": container with ID starting with 6eab84f42a3b193aa6bb10b5146281bc5d17707e874d850913589fee102da668 not found: ID does not exist"
Sep 09 15:38:46.802010 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.801990    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:38:46.835144 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.835074    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"]
Sep 09 15:38:46.844318 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.844299    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c76a304-599d4d688b-hfr7q"]
Sep 09 15:38:46.867641 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.867606    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a73d2be8-cfc6-4c0f-8552-476bd9bc9534" path="/var/lib/kubelet/pods/a73d2be8-cfc6-4c0f-8552-476bd9bc9534/volumes"
Sep 09 15:38:46.921700 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:46.921667    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:47.309778 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:47.309751    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:47.403634 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:47.403600    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:47.550689 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:47.550652    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:47.904512 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:47.904473    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:48.315363 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:48.315285    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:50.025999 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:50.025952    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:50.204282 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:50.204254    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:50.307550 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:50.307466    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:50.869098 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:50.869064    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"]
Sep 09 15:38:50.869661 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:50.869645    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a73d2be8-cfc6-4c0f-8552-476bd9bc9534" containerName="maas-api"
Sep 09 15:38:50.869708 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:50.869665    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a73d2be8-cfc6-4c0f-8552-476bd9bc9534" containerName="maas-api"
Sep 09 15:38:50.869768 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:50.869757    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a73d2be8-cfc6-4c0f-8552-476bd9bc9534" containerName="maas-api"
Sep 09 15:38:50.874469 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:50.874446    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:38:50.876831 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:50.876811    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-a7ffc761-serving-cert\""
Sep 09 15:38:50.884811 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:50.884786    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"]
Sep 09 15:38:50.916430 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:50.916387    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:50.970742 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:50.970714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1926b0b-9710-4109-8aeb-caaf6ec79246-metrics-certs\") pod \"maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg\" (UID: \"c1926b0b-9710-4109-8aeb-caaf6ec79246\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:38:50.970998 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:50.970843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1926b0b-9710-4109-8aeb-caaf6ec79246-openshift-service-ca\") pod \"maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg\" (UID: \"c1926b0b-9710-4109-8aeb-caaf6ec79246\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:38:50.971270 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:50.971250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1926b0b-9710-4109-8aeb-caaf6ec79246-maas-api-tls\") pod \"maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg\" (UID: \"c1926b0b-9710-4109-8aeb-caaf6ec79246\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:38:50.971348 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:50.971291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8hfz\" (UniqueName: \"kubernetes.io/projected/c1926b0b-9710-4109-8aeb-caaf6ec79246-kube-api-access-j8hfz\") pod \"maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg\" (UID: \"c1926b0b-9710-4109-8aeb-caaf6ec79246\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:38:51.003154 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:51.003118    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:51.072433 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:51.072405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1926b0b-9710-4109-8aeb-caaf6ec79246-openshift-service-ca\") pod \"maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg\" (UID: \"c1926b0b-9710-4109-8aeb-caaf6ec79246\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:38:51.072898 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:51.072521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1926b0b-9710-4109-8aeb-caaf6ec79246-maas-api-tls\") pod \"maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg\" (UID: \"c1926b0b-9710-4109-8aeb-caaf6ec79246\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:38:51.072898 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:51.072562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8hfz\" (UniqueName: \"kubernetes.io/projected/c1926b0b-9710-4109-8aeb-caaf6ec79246-kube-api-access-j8hfz\") pod \"maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg\" (UID: \"c1926b0b-9710-4109-8aeb-caaf6ec79246\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:38:51.072898 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:51.072587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1926b0b-9710-4109-8aeb-caaf6ec79246-metrics-certs\") pod \"maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg\" (UID: \"c1926b0b-9710-4109-8aeb-caaf6ec79246\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:38:51.072898 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:38:51.072683    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-serving-cert: secret "maas-api-e2e-migrate-a7ffc761-serving-cert" not found
Sep 09 15:38:51.072898 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:38:51.072770    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1926b0b-9710-4109-8aeb-caaf6ec79246-maas-api-tls podName:c1926b0b-9710-4109-8aeb-caaf6ec79246 nodeName:}" failed. No retries permitted until 2026-09-09 15:38:51.572750491 +0000 UTC m=+1121.214792475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1926b0b-9710-4109-8aeb-caaf6ec79246-maas-api-tls") pod "maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg" (UID: "c1926b0b-9710-4109-8aeb-caaf6ec79246") : secret "maas-api-e2e-migrate-a7ffc761-serving-cert" not found
Sep 09 15:38:51.073197 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:51.073178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1926b0b-9710-4109-8aeb-caaf6ec79246-openshift-service-ca\") pod \"maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg\" (UID: \"c1926b0b-9710-4109-8aeb-caaf6ec79246\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:38:51.075172 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:51.075147    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1926b0b-9710-4109-8aeb-caaf6ec79246-metrics-certs\") pod \"maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg\" (UID: \"c1926b0b-9710-4109-8aeb-caaf6ec79246\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:38:51.081794 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:51.081770    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8hfz\" (UniqueName: \"kubernetes.io/projected/c1926b0b-9710-4109-8aeb-caaf6ec79246-kube-api-access-j8hfz\") pod \"maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg\" (UID: \"c1926b0b-9710-4109-8aeb-caaf6ec79246\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:38:51.164707 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:51.164681    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:51.526657 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:51.526573    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:51.578385 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:51.578349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1926b0b-9710-4109-8aeb-caaf6ec79246-maas-api-tls\") pod \"maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg\" (UID: \"c1926b0b-9710-4109-8aeb-caaf6ec79246\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:38:51.581086 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:51.581060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1926b0b-9710-4109-8aeb-caaf6ec79246-maas-api-tls\") pod \"maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg\" (UID: \"c1926b0b-9710-4109-8aeb-caaf6ec79246\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:38:51.623633 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:51.623600    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:51.787467 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:51.787385    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:38:52.068659 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:52.068630    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"]
Sep 09 15:38:52.070078 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:38:52.070052    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1926b0b_9710_4109_8aeb_caaf6ec79246.slice/crio-eb2031c085a40ecb2aba69cb8f1ff17cc7562532fe2572f791d14bd6b8800900 WatchSource:0}: Error finding container eb2031c085a40ecb2aba69cb8f1ff17cc7562532fe2572f791d14bd6b8800900: Status 404 returned error can't find the container with id eb2031c085a40ecb2aba69cb8f1ff17cc7562532fe2572f791d14bd6b8800900
Sep 09 15:38:52.212343 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:52.212307    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:52.663832 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:52.663803    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:52.829148 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:52.829119    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg" event={"ID":"c1926b0b-9710-4109-8aeb-caaf6ec79246","Type":"ContainerStarted","Data":"eb2031c085a40ecb2aba69cb8f1ff17cc7562532fe2572f791d14bd6b8800900"}
Sep 09 15:38:52.895783 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:52.895750    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:38:55.847310 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:55.847274    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg" event={"ID":"c1926b0b-9710-4109-8aeb-caaf6ec79246","Type":"ContainerStarted","Data":"93f5c609482b70595a56527b6680a25ad0716afebe62173091e86181cf5b6fef"}
Sep 09 15:38:55.847785 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:55.847393    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:38:55.867254 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:38:55.867147    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg" podStartSLOduration=2.637649327 podStartE2EDuration="5.867128179s" totalImagesPullingTime="3.229478852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:38:55.866749442 +0000 UTC m=+1125.508791432" watchObservedRunningTime="2026-09-09 15:38:55.867128179 +0000 UTC m=+1125.509170168"
Sep 09 15:39:01.858744 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:01.858710    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:39:11.364885 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:11.364855    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:20.964168 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:20.964131    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"]
Sep 09 15:39:20.964695 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:20.964419    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg" podUID="c1926b0b-9710-4109-8aeb-caaf6ec79246" containerName="maas-api" containerID="cri-o://93f5c609482b70595a56527b6680a25ad0716afebe62173091e86181cf5b6fef" gracePeriod=30
Sep 09 15:39:20.993179 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:20.993153    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:21.062313 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.062283    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:21.130335 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.130294    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:21.188631 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.188594    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:21.213014 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.212974    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:39:21.275649 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.275556    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1926b0b-9710-4109-8aeb-caaf6ec79246-openshift-service-ca\") pod \"c1926b0b-9710-4109-8aeb-caaf6ec79246\" (UID: \"c1926b0b-9710-4109-8aeb-caaf6ec79246\") "
Sep 09 15:39:21.275804 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.275668    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8hfz\" (UniqueName: \"kubernetes.io/projected/c1926b0b-9710-4109-8aeb-caaf6ec79246-kube-api-access-j8hfz\") pod \"c1926b0b-9710-4109-8aeb-caaf6ec79246\" (UID: \"c1926b0b-9710-4109-8aeb-caaf6ec79246\") "
Sep 09 15:39:21.275804 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.275754    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1926b0b-9710-4109-8aeb-caaf6ec79246-metrics-certs\") pod \"c1926b0b-9710-4109-8aeb-caaf6ec79246\" (UID: \"c1926b0b-9710-4109-8aeb-caaf6ec79246\") "
Sep 09 15:39:21.275804 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.275779    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1926b0b-9710-4109-8aeb-caaf6ec79246-maas-api-tls\") pod \"c1926b0b-9710-4109-8aeb-caaf6ec79246\" (UID: \"c1926b0b-9710-4109-8aeb-caaf6ec79246\") "
Sep 09 15:39:21.276008 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.275921    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1926b0b-9710-4109-8aeb-caaf6ec79246-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c1926b0b-9710-4109-8aeb-caaf6ec79246" (UID: "c1926b0b-9710-4109-8aeb-caaf6ec79246"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:39:21.276209 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.276150    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1926b0b-9710-4109-8aeb-caaf6ec79246-openshift-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:39:21.279150 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.279119    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1926b0b-9710-4109-8aeb-caaf6ec79246-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c1926b0b-9710-4109-8aeb-caaf6ec79246" (UID: "c1926b0b-9710-4109-8aeb-caaf6ec79246"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:39:21.280943 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.280905    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1926b0b-9710-4109-8aeb-caaf6ec79246-kube-api-access-j8hfz" (OuterVolumeSpecName: "kube-api-access-j8hfz") pod "c1926b0b-9710-4109-8aeb-caaf6ec79246" (UID: "c1926b0b-9710-4109-8aeb-caaf6ec79246"). InnerVolumeSpecName "kube-api-access-j8hfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:39:21.281142 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.281123    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1926b0b-9710-4109-8aeb-caaf6ec79246-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c1926b0b-9710-4109-8aeb-caaf6ec79246" (UID: "c1926b0b-9710-4109-8aeb-caaf6ec79246"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:39:21.314115 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.314086    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:21.377287 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.377255    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8hfz\" (UniqueName: \"kubernetes.io/projected/c1926b0b-9710-4109-8aeb-caaf6ec79246-kube-api-access-j8hfz\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:39:21.377287 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.377285    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1926b0b-9710-4109-8aeb-caaf6ec79246-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:39:21.377287 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.377294    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1926b0b-9710-4109-8aeb-caaf6ec79246-maas-api-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:39:21.779486 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.779454    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:21.990993 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.990946    2578 generic.go:356] "Generic (PLEG): container finished" podID="c1926b0b-9710-4109-8aeb-caaf6ec79246" containerID="93f5c609482b70595a56527b6680a25ad0716afebe62173091e86181cf5b6fef" exitCode=0
Sep 09 15:39:21.991428 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.991002    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg" event={"ID":"c1926b0b-9710-4109-8aeb-caaf6ec79246","Type":"ContainerDied","Data":"93f5c609482b70595a56527b6680a25ad0716afebe62173091e86181cf5b6fef"}
Sep 09 15:39:21.991428 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.991005    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"
Sep 09 15:39:21.991428 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.991042    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg" event={"ID":"c1926b0b-9710-4109-8aeb-caaf6ec79246","Type":"ContainerDied","Data":"eb2031c085a40ecb2aba69cb8f1ff17cc7562532fe2572f791d14bd6b8800900"}
Sep 09 15:39:21.991428 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:21.991063    2578 scope.go:117] "RemoveContainer" containerID="93f5c609482b70595a56527b6680a25ad0716afebe62173091e86181cf5b6fef"
Sep 09 15:39:22.000772 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:22.000755    2578 scope.go:117] "RemoveContainer" containerID="93f5c609482b70595a56527b6680a25ad0716afebe62173091e86181cf5b6fef"
Sep 09 15:39:22.001049 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:39:22.001023    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93f5c609482b70595a56527b6680a25ad0716afebe62173091e86181cf5b6fef\": container with ID starting with 93f5c609482b70595a56527b6680a25ad0716afebe62173091e86181cf5b6fef not found: ID does not exist" containerID="93f5c609482b70595a56527b6680a25ad0716afebe62173091e86181cf5b6fef"
Sep 09 15:39:22.001147 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:22.001054    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f5c609482b70595a56527b6680a25ad0716afebe62173091e86181cf5b6fef"} err="failed to get container status \"93f5c609482b70595a56527b6680a25ad0716afebe62173091e86181cf5b6fef\": rpc error: code = NotFound desc = could not find container \"93f5c609482b70595a56527b6680a25ad0716afebe62173091e86181cf5b6fef\": container with ID starting with 93f5c609482b70595a56527b6680a25ad0716afebe62173091e86181cf5b6fef not found: ID does not exist"
Sep 09 15:39:22.011549 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:22.011528    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"]
Sep 09 15:39:22.016197 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:22.016177    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a7ffc761-7f5cdbc7c9-ktdwg"]
Sep 09 15:39:22.102547 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:22.102473    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:22.509408 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:22.509373    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:22.869709 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:22.869614    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1926b0b-9710-4109-8aeb-caaf6ec79246" path="/var/lib/kubelet/pods/c1926b0b-9710-4109-8aeb-caaf6ec79246/volumes"
Sep 09 15:39:24.258474 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:24.258440    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:24.358742 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:24.358709    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:25.441300 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.441268    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"]
Sep 09 15:39:25.442025 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.442006    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1926b0b-9710-4109-8aeb-caaf6ec79246" containerName="maas-api"
Sep 09 15:39:25.442070 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.442029    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1926b0b-9710-4109-8aeb-caaf6ec79246" containerName="maas-api"
Sep 09 15:39:25.442197 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.442183    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1926b0b-9710-4109-8aeb-caaf6ec79246" containerName="maas-api"
Sep 09 15:39:25.447036 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.447014    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:25.449091 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.449065    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f1fbd128-serving-cert\""
Sep 09 15:39:25.458368 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.458344    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"]
Sep 09 15:39:25.476953 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.476921    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:25.518415 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.518376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edefcf90-99d5-4eb8-8367-4c80563ec632-metrics-certs\") pod \"maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn\" (UID: \"edefcf90-99d5-4eb8-8367-4c80563ec632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:25.518415 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.518413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmchr\" (UniqueName: \"kubernetes.io/projected/edefcf90-99d5-4eb8-8367-4c80563ec632-kube-api-access-vmchr\") pod \"maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn\" (UID: \"edefcf90-99d5-4eb8-8367-4c80563ec632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:25.518654 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.518477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edefcf90-99d5-4eb8-8367-4c80563ec632-maas-api-tls\") pod \"maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn\" (UID: \"edefcf90-99d5-4eb8-8367-4c80563ec632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:25.518654 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.518552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edefcf90-99d5-4eb8-8367-4c80563ec632-openshift-service-ca\") pod \"maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn\" (UID: \"edefcf90-99d5-4eb8-8367-4c80563ec632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:25.534255 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.534224    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:25.619963 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.619920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edefcf90-99d5-4eb8-8367-4c80563ec632-maas-api-tls\") pod \"maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn\" (UID: \"edefcf90-99d5-4eb8-8367-4c80563ec632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:25.620191 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.619972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edefcf90-99d5-4eb8-8367-4c80563ec632-openshift-service-ca\") pod \"maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn\" (UID: \"edefcf90-99d5-4eb8-8367-4c80563ec632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:25.620191 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:39:25.620098    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-serving-cert: secret "maas-api-e2e-ait-f1fbd128-serving-cert" not found
Sep 09 15:39:25.620191 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.620110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edefcf90-99d5-4eb8-8367-4c80563ec632-metrics-certs\") pod \"maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn\" (UID: \"edefcf90-99d5-4eb8-8367-4c80563ec632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:25.620191 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.620137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmchr\" (UniqueName: \"kubernetes.io/projected/edefcf90-99d5-4eb8-8367-4c80563ec632-kube-api-access-vmchr\") pod \"maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn\" (UID: \"edefcf90-99d5-4eb8-8367-4c80563ec632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:25.620191 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:39:25.620171    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edefcf90-99d5-4eb8-8367-4c80563ec632-maas-api-tls podName:edefcf90-99d5-4eb8-8367-4c80563ec632 nodeName:}" failed. No retries permitted until 2026-09-09 15:39:26.120154536 +0000 UTC m=+1155.762196507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/edefcf90-99d5-4eb8-8367-4c80563ec632-maas-api-tls") pod "maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn" (UID: "edefcf90-99d5-4eb8-8367-4c80563ec632") : secret "maas-api-e2e-ait-f1fbd128-serving-cert" not found
Sep 09 15:39:25.620699 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.620677    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edefcf90-99d5-4eb8-8367-4c80563ec632-openshift-service-ca\") pod \"maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn\" (UID: \"edefcf90-99d5-4eb8-8367-4c80563ec632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:25.622647 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.622623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edefcf90-99d5-4eb8-8367-4c80563ec632-metrics-certs\") pod \"maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn\" (UID: \"edefcf90-99d5-4eb8-8367-4c80563ec632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:25.629566 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.629541    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmchr\" (UniqueName: \"kubernetes.io/projected/edefcf90-99d5-4eb8-8367-4c80563ec632-kube-api-access-vmchr\") pod \"maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn\" (UID: \"edefcf90-99d5-4eb8-8367-4c80563ec632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:25.703905 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.703834    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:25.951410 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:25.951376    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:26.035848 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:26.035774    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:26.124926 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:26.124892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edefcf90-99d5-4eb8-8367-4c80563ec632-maas-api-tls\") pod \"maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn\" (UID: \"edefcf90-99d5-4eb8-8367-4c80563ec632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:26.127574 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:26.127551    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edefcf90-99d5-4eb8-8367-4c80563ec632-maas-api-tls\") pod \"maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn\" (UID: \"edefcf90-99d5-4eb8-8367-4c80563ec632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:26.359830 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:26.359739    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:26.633353 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:26.633326    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"]
Sep 09 15:39:26.634086 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:39:26.634050    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedefcf90_99d5_4eb8_8367_4c80563ec632.slice/crio-d31afa89736f188df538ea90acb546609e011b12c53e3a096bc3708bcc41d401 WatchSource:0}: Error finding container d31afa89736f188df538ea90acb546609e011b12c53e3a096bc3708bcc41d401: Status 404 returned error can't find the container with id d31afa89736f188df538ea90acb546609e011b12c53e3a096bc3708bcc41d401
Sep 09 15:39:27.036009 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:27.035953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn" event={"ID":"edefcf90-99d5-4eb8-8367-4c80563ec632","Type":"ContainerStarted","Data":"d31afa89736f188df538ea90acb546609e011b12c53e3a096bc3708bcc41d401"}
Sep 09 15:39:28.042421 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:28.042393    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn" event={"ID":"edefcf90-99d5-4eb8-8367-4c80563ec632","Type":"ContainerStarted","Data":"9e9379a781cdd35c5c6fcac9b2f0dfb46f4388fba980d4b4317485c59655323c"}
Sep 09 15:39:28.042862 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:28.042493    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:28.058451 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:28.058399    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn" podStartSLOduration=2.428774283 podStartE2EDuration="3.058381385s" totalImagesPullingTime="629.607102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:39:28.057866803 +0000 UTC m=+1157.699908792" watchObservedRunningTime="2026-09-09 15:39:28.058381385 +0000 UTC m=+1157.700423374"
Sep 09 15:39:34.019465 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.019386    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"]
Sep 09 15:39:34.023281 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.023254    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.026249 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.026221    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-8a43b2\"/\"autores-model-8a43b2-kserve-self-signed-certs\""
Sep 09 15:39:34.026963 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.026940    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-8a43b2\"/\"kube-root-ca.crt\""
Sep 09 15:39:34.027228 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.027212    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-8a43b2\"/\"openshift-service-ca.crt\""
Sep 09 15:39:34.030254 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.030232    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-8a43b2\"/\"default-dockercfg-5l6v5\""
Sep 09 15:39:34.041770 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.041736    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"]
Sep 09 15:39:34.058785 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.058763    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:34.083785 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.083752    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:34.212216 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.212183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29ss5\" (UniqueName: \"kubernetes.io/projected/dc712d8e-72b0-4c20-994c-3aaeded07d1f-kube-api-access-29ss5\") pod \"autores-model-8a43b2-kserve-8f68b9695-89h2x\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") " pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.212393 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.212233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-tmp-dir\") pod \"autores-model-8a43b2-kserve-8f68b9695-89h2x\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") " pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.212393 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.212321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-dshm\") pod \"autores-model-8a43b2-kserve-8f68b9695-89h2x\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") " pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.212393 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.212369    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc712d8e-72b0-4c20-994c-3aaeded07d1f-tls-certs\") pod \"autores-model-8a43b2-kserve-8f68b9695-89h2x\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") " pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.212593 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.212404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-home\") pod \"autores-model-8a43b2-kserve-8f68b9695-89h2x\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") " pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.212593 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.212445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-model-cache\") pod \"autores-model-8a43b2-kserve-8f68b9695-89h2x\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") " pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.314099 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.314010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-tmp-dir\") pod \"autores-model-8a43b2-kserve-8f68b9695-89h2x\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") " pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.314099 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.314074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-dshm\") pod \"autores-model-8a43b2-kserve-8f68b9695-89h2x\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") " pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.314302 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.314110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc712d8e-72b0-4c20-994c-3aaeded07d1f-tls-certs\") pod \"autores-model-8a43b2-kserve-8f68b9695-89h2x\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") " pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.314302 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.314141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-home\") pod \"autores-model-8a43b2-kserve-8f68b9695-89h2x\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") " pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.314302 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.314179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-model-cache\") pod \"autores-model-8a43b2-kserve-8f68b9695-89h2x\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") " pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.314302 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.314276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29ss5\" (UniqueName: \"kubernetes.io/projected/dc712d8e-72b0-4c20-994c-3aaeded07d1f-kube-api-access-29ss5\") pod \"autores-model-8a43b2-kserve-8f68b9695-89h2x\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") " pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.314496 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.314474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-tmp-dir\") pod \"autores-model-8a43b2-kserve-8f68b9695-89h2x\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") " pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.314568 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.314538    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-home\") pod \"autores-model-8a43b2-kserve-8f68b9695-89h2x\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") " pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.314625 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.314599    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-model-cache\") pod \"autores-model-8a43b2-kserve-8f68b9695-89h2x\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") " pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.316548 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.316522    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-dshm\") pod \"autores-model-8a43b2-kserve-8f68b9695-89h2x\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") " pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.316844 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.316826    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc712d8e-72b0-4c20-994c-3aaeded07d1f-tls-certs\") pod \"autores-model-8a43b2-kserve-8f68b9695-89h2x\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") " pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.321521 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.321497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29ss5\" (UniqueName: \"kubernetes.io/projected/dc712d8e-72b0-4c20-994c-3aaeded07d1f-kube-api-access-29ss5\") pod \"autores-model-8a43b2-kserve-8f68b9695-89h2x\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") " pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.340586 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.340559    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:34.611714 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:34.611688    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"]
Sep 09 15:39:34.612962 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:39:34.612937    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc712d8e_72b0_4c20_994c_3aaeded07d1f.slice/crio-359d2060a5715f0f6c9b01fb27dcbfa28e05630280d6720c9040b6630bea314b WatchSource:0}: Error finding container 359d2060a5715f0f6c9b01fb27dcbfa28e05630280d6720c9040b6630bea314b: Status 404 returned error can't find the container with id 359d2060a5715f0f6c9b01fb27dcbfa28e05630280d6720c9040b6630bea314b
Sep 09 15:39:35.082418 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:35.082383    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x" event={"ID":"dc712d8e-72b0-4c20-994c-3aaeded07d1f","Type":"ContainerStarted","Data":"25c1443a1f3025169fb41331d44c3c4801348534dca67162aa61c5a6ef273a21"}
Sep 09 15:39:35.082801 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:35.082433    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x" event={"ID":"dc712d8e-72b0-4c20-994c-3aaeded07d1f","Type":"ContainerStarted","Data":"359d2060a5715f0f6c9b01fb27dcbfa28e05630280d6720c9040b6630bea314b"}
Sep 09 15:39:35.100069 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:35.100018    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x" podStartSLOduration=2.099997416 podStartE2EDuration="2.099997416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:39:35.097607385 +0000 UTC m=+1164.739649394" watchObservedRunningTime="2026-09-09 15:39:35.099997416 +0000 UTC m=+1164.742039402"
Sep 09 15:39:44.341273 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:44.341239    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:44.341717 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:44.341535    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:44.353762 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:44.353739    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:45.145198 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:45.145170    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:39:51.335727 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:51.335697    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:55.956412 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:55.956373    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"]
Sep 09 15:39:55.956837 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:55.956629    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn" podUID="edefcf90-99d5-4eb8-8367-4c80563ec632" containerName="maas-api" containerID="cri-o://9e9379a781cdd35c5c6fcac9b2f0dfb46f4388fba980d4b4317485c59655323c" gracePeriod=30
Sep 09 15:39:55.983411 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:55.983386    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:56.195635 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:56.195604    2578 generic.go:356] "Generic (PLEG): container finished" podID="edefcf90-99d5-4eb8-8367-4c80563ec632" containerID="9e9379a781cdd35c5c6fcac9b2f0dfb46f4388fba980d4b4317485c59655323c" exitCode=0
Sep 09 15:39:56.195804 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:56.195685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn" event={"ID":"edefcf90-99d5-4eb8-8367-4c80563ec632","Type":"ContainerDied","Data":"9e9379a781cdd35c5c6fcac9b2f0dfb46f4388fba980d4b4317485c59655323c"}
Sep 09 15:39:56.220494 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:56.220440    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:56.358335 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:56.358302    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edefcf90-99d5-4eb8-8367-4c80563ec632-maas-api-tls\") pod \"edefcf90-99d5-4eb8-8367-4c80563ec632\" (UID: \"edefcf90-99d5-4eb8-8367-4c80563ec632\") "
Sep 09 15:39:56.358504 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:56.358342    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edefcf90-99d5-4eb8-8367-4c80563ec632-openshift-service-ca\") pod \"edefcf90-99d5-4eb8-8367-4c80563ec632\" (UID: \"edefcf90-99d5-4eb8-8367-4c80563ec632\") "
Sep 09 15:39:56.358504 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:56.358376    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmchr\" (UniqueName: \"kubernetes.io/projected/edefcf90-99d5-4eb8-8367-4c80563ec632-kube-api-access-vmchr\") pod \"edefcf90-99d5-4eb8-8367-4c80563ec632\" (UID: \"edefcf90-99d5-4eb8-8367-4c80563ec632\") "
Sep 09 15:39:56.358504 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:56.358425    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edefcf90-99d5-4eb8-8367-4c80563ec632-metrics-certs\") pod \"edefcf90-99d5-4eb8-8367-4c80563ec632\" (UID: \"edefcf90-99d5-4eb8-8367-4c80563ec632\") "
Sep 09 15:39:56.358768 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:56.358740    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edefcf90-99d5-4eb8-8367-4c80563ec632-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "edefcf90-99d5-4eb8-8367-4c80563ec632" (UID: "edefcf90-99d5-4eb8-8367-4c80563ec632"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:39:56.360750 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:56.360724    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edefcf90-99d5-4eb8-8367-4c80563ec632-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "edefcf90-99d5-4eb8-8367-4c80563ec632" (UID: "edefcf90-99d5-4eb8-8367-4c80563ec632"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:39:56.360853 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:56.360748    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edefcf90-99d5-4eb8-8367-4c80563ec632-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "edefcf90-99d5-4eb8-8367-4c80563ec632" (UID: "edefcf90-99d5-4eb8-8367-4c80563ec632"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:39:56.360853 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:56.360783    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edefcf90-99d5-4eb8-8367-4c80563ec632-kube-api-access-vmchr" (OuterVolumeSpecName: "kube-api-access-vmchr") pod "edefcf90-99d5-4eb8-8367-4c80563ec632" (UID: "edefcf90-99d5-4eb8-8367-4c80563ec632"). InnerVolumeSpecName "kube-api-access-vmchr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:39:56.459765 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:56.459730    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmchr\" (UniqueName: \"kubernetes.io/projected/edefcf90-99d5-4eb8-8367-4c80563ec632-kube-api-access-vmchr\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:39:56.459765 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:56.459760    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edefcf90-99d5-4eb8-8367-4c80563ec632-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:39:56.459765 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:56.459770    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edefcf90-99d5-4eb8-8367-4c80563ec632-maas-api-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:39:56.460023 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:56.459777    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edefcf90-99d5-4eb8-8367-4c80563ec632-openshift-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:39:57.203904 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:57.203874    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"
Sep 09 15:39:57.204411 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:57.203873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn" event={"ID":"edefcf90-99d5-4eb8-8367-4c80563ec632","Type":"ContainerDied","Data":"d31afa89736f188df538ea90acb546609e011b12c53e3a096bc3708bcc41d401"}
Sep 09 15:39:57.204411 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:57.204032    2578 scope.go:117] "RemoveContainer" containerID="9e9379a781cdd35c5c6fcac9b2f0dfb46f4388fba980d4b4317485c59655323c"
Sep 09 15:39:57.220880 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:57.220856    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"]
Sep 09 15:39:57.226684 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:57.226662    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f1fbd128-6ddff864cc-rs7vn"]
Sep 09 15:39:57.670683 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:57.670656    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:57.851742 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:57.851712    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:58.168223 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:58.168185    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:58.638424 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:58.638396    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:39:58.867835 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:39:58.867802    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edefcf90-99d5-4eb8-8367-4c80563ec632" path="/var/lib/kubelet/pods/edefcf90-99d5-4eb8-8367-4c80563ec632/volumes"
Sep 09 15:40:00.516391 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:00.516360    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:00.641353 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:00.641321    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:01.669646 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:01.669619    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:01.884939 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:01.884902    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:01.987605 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:01.987526    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:02.160526 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:02.160496    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:02.237951 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:02.237859    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:02.791257 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:02.791226    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:03.467398 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:03.467363    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:08.965267 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:08.965235    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:15.141476 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:15.141443    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 15:40:15.142411 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:15.142381    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 15:40:15.150288 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:15.150267    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:40:15.151063 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:15.151045    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:40:17.645816 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:17.645787    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:17.706263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:17.706223    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:19.284807 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:19.284775    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"]
Sep 09 15:40:19.285215 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:19.285034    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d" podUID="c68e5346-e4e9-4e09-af2f-98cb26f3cce7" containerName="maas-api" containerID="cri-o://150d0d22e8cbc48a3996787fbd6bbf180441414d0319a2135bb6bc54a9fa9b6b" gracePeriod=30
Sep 09 15:40:19.509233 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:19.509203    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:19.552960 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:19.552935    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:40:19.573968 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:19.573939    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:19.603297 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:19.603266    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52fkg\" (UniqueName: \"kubernetes.io/projected/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-kube-api-access-52fkg\") pod \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\" (UID: \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\") "
Sep 09 15:40:19.603502 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:19.603318    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-metrics-certs\") pod \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\" (UID: \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\") "
Sep 09 15:40:19.603502 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:19.603344    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-maas-api-tls\") pod \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\" (UID: \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\") "
Sep 09 15:40:19.603502 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:19.603399    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-openshift-service-ca\") pod \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\" (UID: \"c68e5346-e4e9-4e09-af2f-98cb26f3cce7\") "
Sep 09 15:40:19.604069 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:19.604035    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c68e5346-e4e9-4e09-af2f-98cb26f3cce7" (UID: "c68e5346-e4e9-4e09-af2f-98cb26f3cce7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:40:19.606367 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:19.606332    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-kube-api-access-52fkg" (OuterVolumeSpecName: "kube-api-access-52fkg") pod "c68e5346-e4e9-4e09-af2f-98cb26f3cce7" (UID: "c68e5346-e4e9-4e09-af2f-98cb26f3cce7"). InnerVolumeSpecName "kube-api-access-52fkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:40:19.606798 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:19.606768    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c68e5346-e4e9-4e09-af2f-98cb26f3cce7" (UID: "c68e5346-e4e9-4e09-af2f-98cb26f3cce7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:40:19.606912 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:19.606897    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c68e5346-e4e9-4e09-af2f-98cb26f3cce7" (UID: "c68e5346-e4e9-4e09-af2f-98cb26f3cce7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:40:19.705087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:19.705054    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-openshift-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:40:19.705087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:19.705081    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52fkg\" (UniqueName: \"kubernetes.io/projected/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-kube-api-access-52fkg\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:40:19.705087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:19.705091    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:40:19.705300 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:19.705100    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c68e5346-e4e9-4e09-af2f-98cb26f3cce7-maas-api-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:40:20.336421 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.336390    2578 generic.go:356] "Generic (PLEG): container finished" podID="c68e5346-e4e9-4e09-af2f-98cb26f3cce7" containerID="150d0d22e8cbc48a3996787fbd6bbf180441414d0319a2135bb6bc54a9fa9b6b" exitCode=0
Sep 09 15:40:20.336849 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.336438    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"
Sep 09 15:40:20.336849 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.336461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d" event={"ID":"c68e5346-e4e9-4e09-af2f-98cb26f3cce7","Type":"ContainerDied","Data":"150d0d22e8cbc48a3996787fbd6bbf180441414d0319a2135bb6bc54a9fa9b6b"}
Sep 09 15:40:20.336849 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.336502    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d" event={"ID":"c68e5346-e4e9-4e09-af2f-98cb26f3cce7","Type":"ContainerDied","Data":"40219e959635c3776933f6c9b341b0c625e4e19098c9037d627e2065ffeb5d4f"}
Sep 09 15:40:20.336849 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.336523    2578 scope.go:117] "RemoveContainer" containerID="150d0d22e8cbc48a3996787fbd6bbf180441414d0319a2135bb6bc54a9fa9b6b"
Sep 09 15:40:20.346284 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.346269    2578 scope.go:117] "RemoveContainer" containerID="150d0d22e8cbc48a3996787fbd6bbf180441414d0319a2135bb6bc54a9fa9b6b"
Sep 09 15:40:20.346538 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:40:20.346521    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150d0d22e8cbc48a3996787fbd6bbf180441414d0319a2135bb6bc54a9fa9b6b\": container with ID starting with 150d0d22e8cbc48a3996787fbd6bbf180441414d0319a2135bb6bc54a9fa9b6b not found: ID does not exist" containerID="150d0d22e8cbc48a3996787fbd6bbf180441414d0319a2135bb6bc54a9fa9b6b"
Sep 09 15:40:20.346580 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.346544    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150d0d22e8cbc48a3996787fbd6bbf180441414d0319a2135bb6bc54a9fa9b6b"} err="failed to get container status \"150d0d22e8cbc48a3996787fbd6bbf180441414d0319a2135bb6bc54a9fa9b6b\": rpc error: code = NotFound desc = could not find container \"150d0d22e8cbc48a3996787fbd6bbf180441414d0319a2135bb6bc54a9fa9b6b\": container with ID starting with 150d0d22e8cbc48a3996787fbd6bbf180441414d0319a2135bb6bc54a9fa9b6b not found: ID does not exist"
Sep 09 15:40:20.359211 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.359165    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"]
Sep 09 15:40:20.361211 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.361193    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d"]
Sep 09 15:40:20.444475 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.444446    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"]
Sep 09 15:40:20.444887 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.444875    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edefcf90-99d5-4eb8-8367-4c80563ec632" containerName="maas-api"
Sep 09 15:40:20.444939 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.444889    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="edefcf90-99d5-4eb8-8367-4c80563ec632" containerName="maas-api"
Sep 09 15:40:20.444939 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.444899    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c68e5346-e4e9-4e09-af2f-98cb26f3cce7" containerName="maas-api"
Sep 09 15:40:20.444939 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.444903    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c68e5346-e4e9-4e09-af2f-98cb26f3cce7" containerName="maas-api"
Sep 09 15:40:20.445065 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.444958    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c68e5346-e4e9-4e09-af2f-98cb26f3cce7" containerName="maas-api"
Sep 09 15:40:20.445065 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.444969    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="edefcf90-99d5-4eb8-8367-4c80563ec632" containerName="maas-api"
Sep 09 15:40:20.449713 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.449687    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.452100 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.452080    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-57b579\"/\"openshift-service-ca.crt\""
Sep 09 15:40:20.452225 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.452160    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-57b579\"/\"kube-root-ca.crt\""
Sep 09 15:40:20.452319 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.452297    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-57b579\"/\"default-dockercfg-6fw6r\""
Sep 09 15:40:20.452379 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.452338    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-57b579\"/\"test-model-57b579-kserve-self-signed-certs\""
Sep 09 15:40:20.457755 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.457737    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"]
Sep 09 15:40:20.513360 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.513331    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4gsx\" (UniqueName: \"kubernetes.io/projected/0064268b-5817-4b48-8fea-ed07df45693f-kube-api-access-s4gsx\") pod \"test-model-57b579-kserve-7f589c9f9f-w4lll\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") " pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.513509 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.513381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-dshm\") pod \"test-model-57b579-kserve-7f589c9f9f-w4lll\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") " pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.513509 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.513436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-model-cache\") pod \"test-model-57b579-kserve-7f589c9f9f-w4lll\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") " pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.513509 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.513466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-tmp-dir\") pod \"test-model-57b579-kserve-7f589c9f9f-w4lll\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") " pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.513509 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.513492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-home\") pod \"test-model-57b579-kserve-7f589c9f9f-w4lll\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") " pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.513649 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.513559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0064268b-5817-4b48-8fea-ed07df45693f-tls-certs\") pod \"test-model-57b579-kserve-7f589c9f9f-w4lll\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") " pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.614968 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.614885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-home\") pod \"test-model-57b579-kserve-7f589c9f9f-w4lll\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") " pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.614968 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.614956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0064268b-5817-4b48-8fea-ed07df45693f-tls-certs\") pod \"test-model-57b579-kserve-7f589c9f9f-w4lll\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") " pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.615219 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.615048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gsx\" (UniqueName: \"kubernetes.io/projected/0064268b-5817-4b48-8fea-ed07df45693f-kube-api-access-s4gsx\") pod \"test-model-57b579-kserve-7f589c9f9f-w4lll\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") " pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.615219 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.615149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-dshm\") pod \"test-model-57b579-kserve-7f589c9f9f-w4lll\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") " pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.615219 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.615185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-model-cache\") pod \"test-model-57b579-kserve-7f589c9f9f-w4lll\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") " pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.615219 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.615209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-tmp-dir\") pod \"test-model-57b579-kserve-7f589c9f9f-w4lll\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") " pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.615456 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.615268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-home\") pod \"test-model-57b579-kserve-7f589c9f9f-w4lll\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") " pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.615554 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.615533    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-model-cache\") pod \"test-model-57b579-kserve-7f589c9f9f-w4lll\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") " pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.615614 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.615581    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-tmp-dir\") pod \"test-model-57b579-kserve-7f589c9f9f-w4lll\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") " pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.617804 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.617782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-dshm\") pod \"test-model-57b579-kserve-7f589c9f9f-w4lll\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") " pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.617971 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.617950    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0064268b-5817-4b48-8fea-ed07df45693f-tls-certs\") pod \"test-model-57b579-kserve-7f589c9f9f-w4lll\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") " pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.623063 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.623039    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4gsx\" (UniqueName: \"kubernetes.io/projected/0064268b-5817-4b48-8fea-ed07df45693f-kube-api-access-s4gsx\") pod \"test-model-57b579-kserve-7f589c9f9f-w4lll\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") " pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.760288 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.760252    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:20.869428 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:20.869352    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c68e5346-e4e9-4e09-af2f-98cb26f3cce7" path="/var/lib/kubelet/pods/c68e5346-e4e9-4e09-af2f-98cb26f3cce7/volumes"
Sep 09 15:40:21.033597 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:21.033573    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"]
Sep 09 15:40:21.035640 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:40:21.035612    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0064268b_5817_4b48_8fea_ed07df45693f.slice/crio-c20a592380449eafb7465f48995b3cdd7f83aec7748b81ad50c4291bff163afa WatchSource:0}: Error finding container c20a592380449eafb7465f48995b3cdd7f83aec7748b81ad50c4291bff163afa: Status 404 returned error can't find the container with id c20a592380449eafb7465f48995b3cdd7f83aec7748b81ad50c4291bff163afa
Sep 09 15:40:21.344408 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:21.344373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll" event={"ID":"0064268b-5817-4b48-8fea-ed07df45693f","Type":"ContainerStarted","Data":"a508efaa24a68cbd93a90247ffd3da17920b5880c3b02e87940ff50837e7141b"}
Sep 09 15:40:21.344408 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:21.344411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll" event={"ID":"0064268b-5817-4b48-8fea-ed07df45693f","Type":"ContainerStarted","Data":"c20a592380449eafb7465f48995b3cdd7f83aec7748b81ad50c4291bff163afa"}
Sep 09 15:40:21.360107 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:21.360055    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll" podStartSLOduration=1.360036572 podStartE2EDuration="1.360036572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:40:21.359405363 +0000 UTC m=+1211.001447351" watchObservedRunningTime="2026-09-09 15:40:21.360036572 +0000 UTC m=+1211.002078575"
Sep 09 15:40:22.492072 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:22.492021    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-autores-8a43b2-7dd8464c4-d686d" podUID="c68e5346-e4e9-4e09-af2f-98cb26f3cce7" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.50:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 09 15:40:22.884642 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:22.884606    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:22.985416 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:22.985383    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:24.847711 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:24.847679    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"]
Sep 09 15:40:24.887381 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:24.887354    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:30.760862 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:30.760830    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:30.761303 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:30.760893    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:30.773458 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:30.773434    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:31.410217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:31.410188    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:40:32.200801 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:32.200760    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:32.272289 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:32.272250    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:32.372293 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:32.372267    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:32.435477 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:32.435423    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:32.552393 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:32.552315    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:33.274002 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:33.273945    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:33.370607 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:33.370576    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:33.470333 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:33.470306    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:34.857053 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:34.857026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:39.553013 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:39.552965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:39.869061 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:39.868963    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x" podUID="dc712d8e-72b0-4c20-994c-3aaeded07d1f" containerName="main" containerID="cri-o://25c1443a1f3025169fb41331d44c3c4801348534dca67162aa61c5a6ef273a21" gracePeriod=45
Sep 09 15:40:40.121135 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.121079    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:40:40.315093 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.315064    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-home\") pod \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") "
Sep 09 15:40:40.315290 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.315173    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-tmp-dir\") pod \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") "
Sep 09 15:40:40.315290 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.315206    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-dshm\") pod \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") "
Sep 09 15:40:40.315290 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.315254    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29ss5\" (UniqueName: \"kubernetes.io/projected/dc712d8e-72b0-4c20-994c-3aaeded07d1f-kube-api-access-29ss5\") pod \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") "
Sep 09 15:40:40.315453 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.315290    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc712d8e-72b0-4c20-994c-3aaeded07d1f-tls-certs\") pod \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") "
Sep 09 15:40:40.315453 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.315353    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-home" (OuterVolumeSpecName: "home") pod "dc712d8e-72b0-4c20-994c-3aaeded07d1f" (UID: "dc712d8e-72b0-4c20-994c-3aaeded07d1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:40:40.315453 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.315390    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-model-cache\") pod \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\" (UID: \"dc712d8e-72b0-4c20-994c-3aaeded07d1f\") "
Sep 09 15:40:40.315453 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.315428    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc712d8e-72b0-4c20-994c-3aaeded07d1f" (UID: "dc712d8e-72b0-4c20-994c-3aaeded07d1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:40:40.315727 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.315612    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc712d8e-72b0-4c20-994c-3aaeded07d1f" (UID: "dc712d8e-72b0-4c20-994c-3aaeded07d1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:40:40.315786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.315730    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-model-cache\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:40:40.315786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.315748    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-home\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:40:40.315786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.315762    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-tmp-dir\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:40:40.317559 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.317537    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-dshm" (OuterVolumeSpecName: "dshm") pod "dc712d8e-72b0-4c20-994c-3aaeded07d1f" (UID: "dc712d8e-72b0-4c20-994c-3aaeded07d1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:40:40.317649 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.317593    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc712d8e-72b0-4c20-994c-3aaeded07d1f-kube-api-access-29ss5" (OuterVolumeSpecName: "kube-api-access-29ss5") pod "dc712d8e-72b0-4c20-994c-3aaeded07d1f" (UID: "dc712d8e-72b0-4c20-994c-3aaeded07d1f"). InnerVolumeSpecName "kube-api-access-29ss5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:40:40.317649 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.317616    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc712d8e-72b0-4c20-994c-3aaeded07d1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc712d8e-72b0-4c20-994c-3aaeded07d1f" (UID: "dc712d8e-72b0-4c20-994c-3aaeded07d1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:40:40.416840 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.416811    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc712d8e-72b0-4c20-994c-3aaeded07d1f-dshm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:40:40.416840 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.416841    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29ss5\" (UniqueName: \"kubernetes.io/projected/dc712d8e-72b0-4c20-994c-3aaeded07d1f-kube-api-access-29ss5\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:40:40.417083 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.416856    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc712d8e-72b0-4c20-994c-3aaeded07d1f-tls-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:40:40.449191 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.449160    2578 generic.go:356] "Generic (PLEG): container finished" podID="dc712d8e-72b0-4c20-994c-3aaeded07d1f" containerID="25c1443a1f3025169fb41331d44c3c4801348534dca67162aa61c5a6ef273a21" exitCode=0
Sep 09 15:40:40.449334 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.449209    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"
Sep 09 15:40:40.449334 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.449249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x" event={"ID":"dc712d8e-72b0-4c20-994c-3aaeded07d1f","Type":"ContainerDied","Data":"25c1443a1f3025169fb41331d44c3c4801348534dca67162aa61c5a6ef273a21"}
Sep 09 15:40:40.449334 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.449293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x" event={"ID":"dc712d8e-72b0-4c20-994c-3aaeded07d1f","Type":"ContainerDied","Data":"359d2060a5715f0f6c9b01fb27dcbfa28e05630280d6720c9040b6630bea314b"}
Sep 09 15:40:40.449334 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.449312    2578 scope.go:117] "RemoveContainer" containerID="25c1443a1f3025169fb41331d44c3c4801348534dca67162aa61c5a6ef273a21"
Sep 09 15:40:40.460812 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.460623    2578 scope.go:117] "RemoveContainer" containerID="25c1443a1f3025169fb41331d44c3c4801348534dca67162aa61c5a6ef273a21"
Sep 09 15:40:40.461396 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:40:40.461255    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c1443a1f3025169fb41331d44c3c4801348534dca67162aa61c5a6ef273a21\": container with ID starting with 25c1443a1f3025169fb41331d44c3c4801348534dca67162aa61c5a6ef273a21 not found: ID does not exist" containerID="25c1443a1f3025169fb41331d44c3c4801348534dca67162aa61c5a6ef273a21"
Sep 09 15:40:40.461396 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.461289    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c1443a1f3025169fb41331d44c3c4801348534dca67162aa61c5a6ef273a21"} err="failed to get container status \"25c1443a1f3025169fb41331d44c3c4801348534dca67162aa61c5a6ef273a21\": rpc error: code = NotFound desc = could not find container \"25c1443a1f3025169fb41331d44c3c4801348534dca67162aa61c5a6ef273a21\": container with ID starting with 25c1443a1f3025169fb41331d44c3c4801348534dca67162aa61c5a6ef273a21 not found: ID does not exist"
Sep 09 15:40:40.470673 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.470649    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"]
Sep 09 15:40:40.474600 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.474579    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-8a43b2/autores-model-8a43b2-kserve-8f68b9695-89h2x"]
Sep 09 15:40:40.551916 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.551887    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:40.661584 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.661557    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:40.753633 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.753565    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:40.867691 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.867660    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc712d8e-72b0-4c20-994c-3aaeded07d1f" path="/var/lib/kubelet/pods/dc712d8e-72b0-4c20-994c-3aaeded07d1f/volumes"
Sep 09 15:40:40.962366 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:40.962332    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:41.270628 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:41.270601    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:43.457537 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.457505    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"]
Sep 09 15:40:43.457972 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.457957    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc712d8e-72b0-4c20-994c-3aaeded07d1f" containerName="main"
Sep 09 15:40:43.457972 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.457974    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc712d8e-72b0-4c20-994c-3aaeded07d1f" containerName="main"
Sep 09 15:40:43.458087 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.458065    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc712d8e-72b0-4c20-994c-3aaeded07d1f" containerName="main"
Sep 09 15:40:43.462292 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.462271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.464362 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.464336    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-df30be\"/\"premium-w0-df30be-kserve-self-signed-certs\""
Sep 09 15:40:43.468679 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.468656    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"]
Sep 09 15:40:43.546422 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.546393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/861de506-45f1-4566-a610-ecf1f980202f-tls-certs\") pod \"premium-w0-df30be-kserve-6d79b86fc6-hpzmv\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") " pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.546568 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.546448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-home\") pod \"premium-w0-df30be-kserve-6d79b86fc6-hpzmv\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") " pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.546618 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.546595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-dshm\") pod \"premium-w0-df30be-kserve-6d79b86fc6-hpzmv\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") " pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.546667 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.546651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-tmp-dir\") pod \"premium-w0-df30be-kserve-6d79b86fc6-hpzmv\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") " pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.546715 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.546699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjmnp\" (UniqueName: \"kubernetes.io/projected/861de506-45f1-4566-a610-ecf1f980202f-kube-api-access-gjmnp\") pod \"premium-w0-df30be-kserve-6d79b86fc6-hpzmv\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") " pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.546757 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.546744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-model-cache\") pod \"premium-w0-df30be-kserve-6d79b86fc6-hpzmv\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") " pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.557244 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.557214    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:40:43.648614 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.648532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-dshm\") pod \"premium-w0-df30be-kserve-6d79b86fc6-hpzmv\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") " pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.648614 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.648598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-tmp-dir\") pod \"premium-w0-df30be-kserve-6d79b86fc6-hpzmv\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") " pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.649021 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.648647    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjmnp\" (UniqueName: \"kubernetes.io/projected/861de506-45f1-4566-a610-ecf1f980202f-kube-api-access-gjmnp\") pod \"premium-w0-df30be-kserve-6d79b86fc6-hpzmv\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") " pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.649021 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.648687    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-model-cache\") pod \"premium-w0-df30be-kserve-6d79b86fc6-hpzmv\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") " pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.649021 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.648730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/861de506-45f1-4566-a610-ecf1f980202f-tls-certs\") pod \"premium-w0-df30be-kserve-6d79b86fc6-hpzmv\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") " pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.649021 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.648782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-home\") pod \"premium-w0-df30be-kserve-6d79b86fc6-hpzmv\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") " pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.649453 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.649264    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-home\") pod \"premium-w0-df30be-kserve-6d79b86fc6-hpzmv\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") " pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.649577 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.649536    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-tmp-dir\") pod \"premium-w0-df30be-kserve-6d79b86fc6-hpzmv\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") " pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.649961 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.649894    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-model-cache\") pod \"premium-w0-df30be-kserve-6d79b86fc6-hpzmv\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") " pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.652136 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.652096    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-dshm\") pod \"premium-w0-df30be-kserve-6d79b86fc6-hpzmv\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") " pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.652484 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.652456    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/861de506-45f1-4566-a610-ecf1f980202f-tls-certs\") pod \"premium-w0-df30be-kserve-6d79b86fc6-hpzmv\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") " pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.657884 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.657862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjmnp\" (UniqueName: \"kubernetes.io/projected/861de506-45f1-4566-a610-ecf1f980202f-kube-api-access-gjmnp\") pod \"premium-w0-df30be-kserve-6d79b86fc6-hpzmv\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") " pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:43.773193 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:43.773107    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:44.049684 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:44.049658    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"]
Sep 09 15:40:44.052466 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:40:44.052430    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod861de506_45f1_4566_a610_ecf1f980202f.slice/crio-0a5b611e3bb879ca78fc7ba4d623c49b0e5acaf60aa0666ea6218b354ade1ea0 WatchSource:0}: Error finding container 0a5b611e3bb879ca78fc7ba4d623c49b0e5acaf60aa0666ea6218b354ade1ea0: Status 404 returned error can't find the container with id 0a5b611e3bb879ca78fc7ba4d623c49b0e5acaf60aa0666ea6218b354ade1ea0
Sep 09 15:40:44.475166 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:44.475134    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv" event={"ID":"861de506-45f1-4566-a610-ecf1f980202f","Type":"ContainerStarted","Data":"97be43e9ea830385e684e81a0ad0b8e6614c0906a377547f5a041777d5184c7e"}
Sep 09 15:40:44.475525 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:44.475175    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv" event={"ID":"861de506-45f1-4566-a610-ecf1f980202f","Type":"ContainerStarted","Data":"0a5b611e3bb879ca78fc7ba4d623c49b0e5acaf60aa0666ea6218b354ade1ea0"}
Sep 09 15:40:44.490219 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:44.490169    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv" podStartSLOduration=1.490149977 podStartE2EDuration="1.490149977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:40:44.490060538 +0000 UTC m=+1234.132102528" watchObservedRunningTime="2026-09-09 15:40:44.490149977 +0000 UTC m=+1234.132191966"
Sep 09 15:40:53.774016 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:53.773962    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:53.774563 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:53.774039    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:53.788207 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:53.788176    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:40:54.538658 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:40:54.538621    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:41:01.740770 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:01.740661    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:02.064011 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:02.063960    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"]
Sep 09 15:41:02.064260 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:02.064209    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq" podUID="0e9550f7-8239-4de0-afe3-c793a9f7fa56" containerName="istio-proxy" containerID="cri-o://b78ce0c46d1dec1da891d43ffd503572b3f1bffe59780c1329f0789a68b99d20" gracePeriod=30
Sep 09 15:41:02.104830 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:02.104802    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:02.542891 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:02.542858    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:02.954932 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:02.954868    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:03.460902 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:03.460871    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:04.808254 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:04.808210    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:04.871343 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:04.871312    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:05.695654 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:05.695624    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:05.789503 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:05.789473    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:07.076768 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.076746    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:07.179341 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.179303    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:07.315868 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.315844    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:41:07.324177 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.324153    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:07.494278 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.494243    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z74z9\" (UniqueName: \"kubernetes.io/projected/0e9550f7-8239-4de0-afe3-c793a9f7fa56-kube-api-access-z74z9\") pod \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") "
Sep 09 15:41:07.494460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.494301    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-token\") pod \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") "
Sep 09 15:41:07.494460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.494350    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-credential-socket\") pod \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") "
Sep 09 15:41:07.494460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.494411    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-podinfo\") pod \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") "
Sep 09 15:41:07.494460 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.494438    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istiod-ca-cert\") pod \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") "
Sep 09 15:41:07.494677 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.494470    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-workload-certs\") pod \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") "
Sep 09 15:41:07.494677 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.494493    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-envoy\") pod \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") "
Sep 09 15:41:07.494677 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.494565    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-data\") pod \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") "
Sep 09 15:41:07.494677 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.494618    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-workload-socket\") pod \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\" (UID: \"0e9550f7-8239-4de0-afe3-c793a9f7fa56\") "
Sep 09 15:41:07.494902 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.494680    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0e9550f7-8239-4de0-afe3-c793a9f7fa56" (UID: "0e9550f7-8239-4de0-afe3-c793a9f7fa56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:41:07.494902 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.494838    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0e9550f7-8239-4de0-afe3-c793a9f7fa56" (UID: "0e9550f7-8239-4de0-afe3-c793a9f7fa56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:41:07.495051 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.495007    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0e9550f7-8239-4de0-afe3-c793a9f7fa56" (UID: "0e9550f7-8239-4de0-afe3-c793a9f7fa56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:41:07.495051 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.495039    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0e9550f7-8239-4de0-afe3-c793a9f7fa56" (UID: "0e9550f7-8239-4de0-afe3-c793a9f7fa56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:41:07.495156 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.495124    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-credential-socket\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:41:07.495156 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.495142    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istiod-ca-cert\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:41:07.495443 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.495157    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-workload-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:41:07.495443 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.495169    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-workload-socket\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:41:07.495443 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.495142    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-data" (OuterVolumeSpecName: "istio-data") pod "0e9550f7-8239-4de0-afe3-c793a9f7fa56" (UID: "0e9550f7-8239-4de0-afe3-c793a9f7fa56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:41:07.496786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.496756    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-token" (OuterVolumeSpecName: "istio-token") pod "0e9550f7-8239-4de0-afe3-c793a9f7fa56" (UID: "0e9550f7-8239-4de0-afe3-c793a9f7fa56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:41:07.496918 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.496896    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0e9550f7-8239-4de0-afe3-c793a9f7fa56" (UID: "0e9550f7-8239-4de0-afe3-c793a9f7fa56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 15:41:07.497231 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.497205    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0e9550f7-8239-4de0-afe3-c793a9f7fa56" (UID: "0e9550f7-8239-4de0-afe3-c793a9f7fa56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:41:07.497231 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.497214    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9550f7-8239-4de0-afe3-c793a9f7fa56-kube-api-access-z74z9" (OuterVolumeSpecName: "kube-api-access-z74z9") pod "0e9550f7-8239-4de0-afe3-c793a9f7fa56" (UID: "0e9550f7-8239-4de0-afe3-c793a9f7fa56"). InnerVolumeSpecName "kube-api-access-z74z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:41:07.580800 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.580769    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:07.595846 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.595814    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-data\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:41:07.595846 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.595842    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z74z9\" (UniqueName: \"kubernetes.io/projected/0e9550f7-8239-4de0-afe3-c793a9f7fa56-kube-api-access-z74z9\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:41:07.596162 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.595859    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-token\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:41:07.596162 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.595875    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-podinfo\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:41:07.596162 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.595887    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e9550f7-8239-4de0-afe3-c793a9f7fa56-istio-envoy\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:41:07.601874 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.601846    2578 generic.go:356] "Generic (PLEG): container finished" podID="0e9550f7-8239-4de0-afe3-c793a9f7fa56" containerID="b78ce0c46d1dec1da891d43ffd503572b3f1bffe59780c1329f0789a68b99d20" exitCode=0
Sep 09 15:41:07.602023 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.601909    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"
Sep 09 15:41:07.602023 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.601931    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq" event={"ID":"0e9550f7-8239-4de0-afe3-c793a9f7fa56","Type":"ContainerDied","Data":"b78ce0c46d1dec1da891d43ffd503572b3f1bffe59780c1329f0789a68b99d20"}
Sep 09 15:41:07.602104 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.602028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq" event={"ID":"0e9550f7-8239-4de0-afe3-c793a9f7fa56","Type":"ContainerDied","Data":"2f4e9ed5eeff4843c27fa49814b3c7bf2e45b7c637cab8977f0b0296fa8071bf"}
Sep 09 15:41:07.602104 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.602063    2578 scope.go:117] "RemoveContainer" containerID="b78ce0c46d1dec1da891d43ffd503572b3f1bffe59780c1329f0789a68b99d20"
Sep 09 15:41:07.612649 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.612580    2578 scope.go:117] "RemoveContainer" containerID="bab3f494e98181d551701c8e54933438b0d051eb6e91eefc24fb7db91e7531ac"
Sep 09 15:41:07.622824 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.622802    2578 scope.go:117] "RemoveContainer" containerID="b78ce0c46d1dec1da891d43ffd503572b3f1bffe59780c1329f0789a68b99d20"
Sep 09 15:41:07.623140 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:41:07.623119    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78ce0c46d1dec1da891d43ffd503572b3f1bffe59780c1329f0789a68b99d20\": container with ID starting with b78ce0c46d1dec1da891d43ffd503572b3f1bffe59780c1329f0789a68b99d20 not found: ID does not exist" containerID="b78ce0c46d1dec1da891d43ffd503572b3f1bffe59780c1329f0789a68b99d20"
Sep 09 15:41:07.623202 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.623152    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78ce0c46d1dec1da891d43ffd503572b3f1bffe59780c1329f0789a68b99d20"} err="failed to get container status \"b78ce0c46d1dec1da891d43ffd503572b3f1bffe59780c1329f0789a68b99d20\": rpc error: code = NotFound desc = could not find container \"b78ce0c46d1dec1da891d43ffd503572b3f1bffe59780c1329f0789a68b99d20\": container with ID starting with b78ce0c46d1dec1da891d43ffd503572b3f1bffe59780c1329f0789a68b99d20 not found: ID does not exist"
Sep 09 15:41:07.623202 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.623169    2578 scope.go:117] "RemoveContainer" containerID="bab3f494e98181d551701c8e54933438b0d051eb6e91eefc24fb7db91e7531ac"
Sep 09 15:41:07.623443 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:41:07.623425    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab3f494e98181d551701c8e54933438b0d051eb6e91eefc24fb7db91e7531ac\": container with ID starting with bab3f494e98181d551701c8e54933438b0d051eb6e91eefc24fb7db91e7531ac not found: ID does not exist" containerID="bab3f494e98181d551701c8e54933438b0d051eb6e91eefc24fb7db91e7531ac"
Sep 09 15:41:07.623502 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.623457    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab3f494e98181d551701c8e54933438b0d051eb6e91eefc24fb7db91e7531ac"} err="failed to get container status \"bab3f494e98181d551701c8e54933438b0d051eb6e91eefc24fb7db91e7531ac\": rpc error: code = NotFound desc = could not find container \"bab3f494e98181d551701c8e54933438b0d051eb6e91eefc24fb7db91e7531ac\": container with ID starting with bab3f494e98181d551701c8e54933438b0d051eb6e91eefc24fb7db91e7531ac not found: ID does not exist"
Sep 09 15:41:07.627200 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.627176    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"]
Sep 09 15:41:07.632720 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.632697    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-8a43b2-openshift-default-5b7d688f58-z5dpq"]
Sep 09 15:41:07.682162 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:07.682133    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:08.868148 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:08.868111    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e9550f7-8239-4de0-afe3-c793a9f7fa56" path="/var/lib/kubelet/pods/0e9550f7-8239-4de0-afe3-c793a9f7fa56/volumes"
Sep 09 15:41:12.862338 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:12.862298    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:12.968538 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:12.968507    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:13.062740 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:13.062714    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:19.364166 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:19.364130    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:24.874110 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:24.874078    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"]
Sep 09 15:41:24.874714 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:24.874694    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e9550f7-8239-4de0-afe3-c793a9f7fa56" containerName="istio-proxy"
Sep 09 15:41:24.874804 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:24.874717    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e9550f7-8239-4de0-afe3-c793a9f7fa56" containerName="istio-proxy"
Sep 09 15:41:24.874860 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:24.874852    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e9550f7-8239-4de0-afe3-c793a9f7fa56" containerName="istio-proxy"
Sep 09 15:41:24.874921 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:24.874871    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e9550f7-8239-4de0-afe3-c793a9f7fa56" containerName="istio-proxy"
Sep 09 15:41:24.877902 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:24.877881    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:24.881261 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:24.881237    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1e15c8\"/\"openshift-service-ca.crt\""
Sep 09 15:41:24.881405 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:24.881333    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1e15c8\"/\"premium-w4-1e15c8-kserve-self-signed-certs\""
Sep 09 15:41:24.881482 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:24.881314    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1e15c8\"/\"kube-root-ca.crt\""
Sep 09 15:41:24.881545 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:24.881290    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1e15c8\"/\"default-dockercfg-qxwlq\""
Sep 09 15:41:24.884968 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:24.884943    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"]
Sep 09 15:41:24.901257 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:24.901232    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:24.972765 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:24.972725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5nvl\" (UniqueName: \"kubernetes.io/projected/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-kube-api-access-v5nvl\") pod \"premium-w4-1e15c8-kserve-8779cb4d9-hdtnf\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") " pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:24.972909 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:24.972783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-home\") pod \"premium-w4-1e15c8-kserve-8779cb4d9-hdtnf\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") " pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:24.972909 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:24.972831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-model-cache\") pod \"premium-w4-1e15c8-kserve-8779cb4d9-hdtnf\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") " pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:24.972909 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:24.972904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-tmp-dir\") pod \"premium-w4-1e15c8-kserve-8779cb4d9-hdtnf\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") " pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:24.973062 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:24.972923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-dshm\") pod \"premium-w4-1e15c8-kserve-8779cb4d9-hdtnf\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") " pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:24.973062 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:24.973028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-tls-certs\") pod \"premium-w4-1e15c8-kserve-8779cb4d9-hdtnf\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") " pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:25.073899 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:25.073865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5nvl\" (UniqueName: \"kubernetes.io/projected/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-kube-api-access-v5nvl\") pod \"premium-w4-1e15c8-kserve-8779cb4d9-hdtnf\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") " pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:25.074090 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:25.073914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-home\") pod \"premium-w4-1e15c8-kserve-8779cb4d9-hdtnf\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") " pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:25.074090 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:25.073933    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-model-cache\") pod \"premium-w4-1e15c8-kserve-8779cb4d9-hdtnf\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") " pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:25.074090 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:25.073969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-tmp-dir\") pod \"premium-w4-1e15c8-kserve-8779cb4d9-hdtnf\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") " pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:25.074090 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:25.074010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-dshm\") pod \"premium-w4-1e15c8-kserve-8779cb4d9-hdtnf\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") " pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:25.074090 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:25.074054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-tls-certs\") pod \"premium-w4-1e15c8-kserve-8779cb4d9-hdtnf\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") " pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:25.074348 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:25.074325    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-home\") pod \"premium-w4-1e15c8-kserve-8779cb4d9-hdtnf\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") " pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:25.074410 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:25.074387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-tmp-dir\") pod \"premium-w4-1e15c8-kserve-8779cb4d9-hdtnf\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") " pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:25.074410 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:25.074399    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-model-cache\") pod \"premium-w4-1e15c8-kserve-8779cb4d9-hdtnf\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") " pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:25.076543 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:25.076517    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-dshm\") pod \"premium-w4-1e15c8-kserve-8779cb4d9-hdtnf\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") " pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:25.076665 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:25.076648    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-tls-certs\") pod \"premium-w4-1e15c8-kserve-8779cb4d9-hdtnf\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") " pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:25.081964 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:25.081940    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5nvl\" (UniqueName: \"kubernetes.io/projected/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-kube-api-access-v5nvl\") pod \"premium-w4-1e15c8-kserve-8779cb4d9-hdtnf\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") " pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:25.188436 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:25.188405    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:25.459033 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:25.459004    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"]
Sep 09 15:41:25.460081 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:41:25.460052    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10a7a5ef_7ff3_42be_862f_8dd4527eecd8.slice/crio-923d6e54b4bc0d25497bd5d09a3cc3d74c6d3dea8071021e0dfd3313cae4e9c9 WatchSource:0}: Error finding container 923d6e54b4bc0d25497bd5d09a3cc3d74c6d3dea8071021e0dfd3313cae4e9c9: Status 404 returned error can't find the container with id 923d6e54b4bc0d25497bd5d09a3cc3d74c6d3dea8071021e0dfd3313cae4e9c9
Sep 09 15:41:25.696051 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:25.696019    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf" event={"ID":"10a7a5ef-7ff3-42be-862f-8dd4527eecd8","Type":"ContainerStarted","Data":"256b6992d791512a312721fbd754ff524299b695d6f1821f778a266cd7e3ff7c"}
Sep 09 15:41:25.696235 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:25.696065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf" event={"ID":"10a7a5ef-7ff3-42be-862f-8dd4527eecd8","Type":"ContainerStarted","Data":"923d6e54b4bc0d25497bd5d09a3cc3d74c6d3dea8071021e0dfd3313cae4e9c9"}
Sep 09 15:41:25.713367 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:25.713280    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf" podStartSLOduration=1.7132641899999999 podStartE2EDuration="1.71326419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:41:25.712111269 +0000 UTC m=+1275.354153252" watchObservedRunningTime="2026-09-09 15:41:25.71326419 +0000 UTC m=+1275.355306192"
Sep 09 15:41:28.967955 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:28.967919    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:30.701253 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:30.701217    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:34.866802 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:34.866765    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:34.967777 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:34.967743    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:35.068530 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:35.068489    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:35.168376 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:35.168320    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:35.188871 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:35.188847    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:35.189009 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:35.188889    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:35.201413 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:35.201391    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:35.760394 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:35.760366    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:41:45.647211 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:45.647174    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:46.441435 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:46.441403    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:47.856022 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:47.855989    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"]
Sep 09 15:41:47.856491 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:47.856474    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e9550f7-8239-4de0-afe3-c793a9f7fa56" containerName="istio-proxy"
Sep 09 15:41:47.856546 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:47.856493    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e9550f7-8239-4de0-afe3-c793a9f7fa56" containerName="istio-proxy"
Sep 09 15:41:47.859662 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:47.859642    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:47.863614 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:47.863540    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-267962\"/\"openshift-service-ca.crt\""
Sep 09 15:41:47.863614 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:47.863579    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-267962\"/\"default-dockercfg-hhkrg\""
Sep 09 15:41:47.863614 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:47.863595    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-267962\"/\"test-model-267962-kserve-self-signed-certs\""
Sep 09 15:41:47.863872 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:47.863845    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-267962\"/\"kube-root-ca.crt\""
Sep 09 15:41:47.870041 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:47.870022    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"]
Sep 09 15:41:47.890679 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:47.890653    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:48.005116 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.005078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-dshm\") pod \"test-model-267962-kserve-c958c5d69-qkpl9\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") " pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.005326 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.005125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqtb8\" (UniqueName: \"kubernetes.io/projected/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-kube-api-access-vqtb8\") pod \"test-model-267962-kserve-c958c5d69-qkpl9\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") " pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.005326 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.005234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-model-cache\") pod \"test-model-267962-kserve-c958c5d69-qkpl9\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") " pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.005326 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.005301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-home\") pod \"test-model-267962-kserve-c958c5d69-qkpl9\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") " pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.005487 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.005342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-tls-certs\") pod \"test-model-267962-kserve-c958c5d69-qkpl9\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") " pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.005487 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.005369    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-tmp-dir\") pod \"test-model-267962-kserve-c958c5d69-qkpl9\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") " pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.106669 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.106575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-dshm\") pod \"test-model-267962-kserve-c958c5d69-qkpl9\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") " pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.106669 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.106630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqtb8\" (UniqueName: \"kubernetes.io/projected/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-kube-api-access-vqtb8\") pod \"test-model-267962-kserve-c958c5d69-qkpl9\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") " pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.106912 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.106682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-model-cache\") pod \"test-model-267962-kserve-c958c5d69-qkpl9\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") " pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.106912 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.106746    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-home\") pod \"test-model-267962-kserve-c958c5d69-qkpl9\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") " pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.106912 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.106795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-tls-certs\") pod \"test-model-267962-kserve-c958c5d69-qkpl9\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") " pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.106912 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.106813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-tmp-dir\") pod \"test-model-267962-kserve-c958c5d69-qkpl9\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") " pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.107166 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.107103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-model-cache\") pod \"test-model-267962-kserve-c958c5d69-qkpl9\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") " pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.107228 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.107168    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-home\") pod \"test-model-267962-kserve-c958c5d69-qkpl9\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") " pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.107281 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.107258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-tmp-dir\") pod \"test-model-267962-kserve-c958c5d69-qkpl9\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") " pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.109043 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.109012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-dshm\") pod \"test-model-267962-kserve-c958c5d69-qkpl9\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") " pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.109487 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.109470    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-tls-certs\") pod \"test-model-267962-kserve-c958c5d69-qkpl9\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") " pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.120698 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.120672    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqtb8\" (UniqueName: \"kubernetes.io/projected/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-kube-api-access-vqtb8\") pod \"test-model-267962-kserve-c958c5d69-qkpl9\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") " pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.170868 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.170830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:48.448515 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.448491    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"]
Sep 09 15:41:48.450394 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:41:48.450369    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7522379_2fe7_483d_b5b9_dafe8f37dcc5.slice/crio-754e8ba3404d729ff20dce059ff2d0a7009b040e9e8780fa63fc62fc04ab6a43 WatchSource:0}: Error finding container 754e8ba3404d729ff20dce059ff2d0a7009b040e9e8780fa63fc62fc04ab6a43: Status 404 returned error can't find the container with id 754e8ba3404d729ff20dce059ff2d0a7009b040e9e8780fa63fc62fc04ab6a43
Sep 09 15:41:48.818738 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.818639    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9" event={"ID":"b7522379-2fe7-483d-b5b9-dafe8f37dcc5","Type":"ContainerStarted","Data":"78794fb17c081c037b4aef81c1018d9edb5da2febb71603943c8af98da22f9c5"}
Sep 09 15:41:48.818738 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.818679    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9" event={"ID":"b7522379-2fe7-483d-b5b9-dafe8f37dcc5","Type":"ContainerStarted","Data":"754e8ba3404d729ff20dce059ff2d0a7009b040e9e8780fa63fc62fc04ab6a43"}
Sep 09 15:41:48.835740 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:48.835691    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9" podStartSLOduration=1.8356745 podStartE2EDuration="1.8356745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:41:48.833947317 +0000 UTC m=+1298.475989297" watchObservedRunningTime="2026-09-09 15:41:48.8356745 +0000 UTC m=+1298.477716489"
Sep 09 15:41:52.087337 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:52.087301    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:52.170180 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:52.170148    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:52.329489 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:52.329459    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:55.618807 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:55.618764    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:55.716422 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:55.716378    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:41:58.171966 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:58.171929    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:58.172467 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:58.172013    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:58.184261 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:58.184233    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:41:58.885727 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:41:58.885700    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:42:01.071674 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:01.071641    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:01.140470 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:01.140438    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:01.727631 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:01.727599    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:07.504464 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:07.504428    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:10.661533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:10.661500    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:10.715579 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:10.715554    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:14.036439 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:14.036408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:14.124913 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:14.124884    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:14.370458 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:14.370369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:15.128233 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.128151    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"]
Sep 09 15:42:15.131927 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.131905    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:15.134158 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.134136    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-73d87788-serving-cert\""
Sep 09 15:42:15.143205 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.143182    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"]
Sep 09 15:42:15.162791 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.162762    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:15.274923 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.274889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tzj7\" (UniqueName: \"kubernetes.io/projected/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-kube-api-access-9tzj7\") pod \"maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz\" (UID: \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:15.275108 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.274945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-metrics-certs\") pod \"maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz\" (UID: \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:15.275108 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.275077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-maas-api-tls\") pod \"maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz\" (UID: \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:15.275108 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.275106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-openshift-service-ca\") pod \"maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz\" (UID: \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:15.287531 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.287499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:15.376259 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.376224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-maas-api-tls\") pod \"maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz\" (UID: \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:15.376259 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.376259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-openshift-service-ca\") pod \"maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz\" (UID: \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:15.376482 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.376338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tzj7\" (UniqueName: \"kubernetes.io/projected/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-kube-api-access-9tzj7\") pod \"maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz\" (UID: \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:15.376482 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:42:15.376357    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-serving-cert: secret "maas-api-e2e-mt-73d87788-serving-cert" not found
Sep 09 15:42:15.376482 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.376376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-metrics-certs\") pod \"maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz\" (UID: \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:15.376482 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:42:15.376432    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-maas-api-tls podName:1a9f3c95-2d2e-456d-a29e-a7eb2b9691da nodeName:}" failed. No retries permitted until 2026-09-09 15:42:15.876408794 +0000 UTC m=+1325.518450774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-maas-api-tls") pod "maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz" (UID: "1a9f3c95-2d2e-456d-a29e-a7eb2b9691da") : secret "maas-api-e2e-mt-73d87788-serving-cert" not found
Sep 09 15:42:15.377026 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.377007    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-openshift-service-ca\") pod \"maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz\" (UID: \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:15.379066 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.379012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-metrics-certs\") pod \"maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz\" (UID: \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:15.389641 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.389606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tzj7\" (UniqueName: \"kubernetes.io/projected/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-kube-api-access-9tzj7\") pod \"maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz\" (UID: \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:15.603741 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.603707    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:15.806774 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.806749    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:15.880708 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.880669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-maas-api-tls\") pod \"maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz\" (UID: \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:15.883400 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.883372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-maas-api-tls\") pod \"maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz\" (UID: \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:15.908745 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:15.908718    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:16.044021 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:16.043991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:16.317663 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:16.317640    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"]
Sep 09 15:42:16.319376 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:42:16.319348    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9f3c95_2d2e_456d_a29e_a7eb2b9691da.slice/crio-53d74948ff5c1adfe60febe05236d0400c6f8c667e90e8c980423ce8bb3b8c43 WatchSource:0}: Error finding container 53d74948ff5c1adfe60febe05236d0400c6f8c667e90e8c980423ce8bb3b8c43: Status 404 returned error can't find the container with id 53d74948ff5c1adfe60febe05236d0400c6f8c667e90e8c980423ce8bb3b8c43
Sep 09 15:42:16.971230 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:16.971194    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz" event={"ID":"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da","Type":"ContainerStarted","Data":"d2d09a2483b0cda4f92ed604bece2a95631f2d62a98d7b90a6e7764d9089e4ac"}
Sep 09 15:42:16.971406 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:16.971254    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz" event={"ID":"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da","Type":"ContainerStarted","Data":"53d74948ff5c1adfe60febe05236d0400c6f8c667e90e8c980423ce8bb3b8c43"}
Sep 09 15:42:16.971406 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:16.971294    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:16.986536 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:16.986478    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz" podStartSLOduration=1.472278653 podStartE2EDuration="1.986463439s" totalImagesPullingTime="514.184786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:42:16.986170462 +0000 UTC m=+1326.628212453" watchObservedRunningTime="2026-09-09 15:42:16.986463439 +0000 UTC m=+1326.628505427"
Sep 09 15:42:20.550063 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:20.550027    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:22.982768 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:22.982738    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:38.981342 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:38.981259    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:39.649630 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:39.649596    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:39.746521 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:39.746493    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:40.044522 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:40.044438    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:45.864562 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:45.864530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:45.925588 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:45.925556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:46.317331 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.317295    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:46.334926 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.334900    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"]
Sep 09 15:42:46.338531 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.338516    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.340723 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.340698    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-df30be\"/\"distinct-w0-df30be-kserve-self-signed-certs\""
Sep 09 15:42:46.347792 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.347769    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"]
Sep 09 15:42:46.491074 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.491046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-home\") pod \"distinct-w0-df30be-kserve-88c5dcd79-qrhq2\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.491242 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.491099    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxl5m\" (UniqueName: \"kubernetes.io/projected/dccfee92-8984-45fd-899b-7ab786c9525b-kube-api-access-rxl5m\") pod \"distinct-w0-df30be-kserve-88c5dcd79-qrhq2\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.491242 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.491229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccfee92-8984-45fd-899b-7ab786c9525b-tls-certs\") pod \"distinct-w0-df30be-kserve-88c5dcd79-qrhq2\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.491321 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.491273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-tmp-dir\") pod \"distinct-w0-df30be-kserve-88c5dcd79-qrhq2\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.491361 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.491332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-dshm\") pod \"distinct-w0-df30be-kserve-88c5dcd79-qrhq2\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.491361 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.491358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-model-cache\") pod \"distinct-w0-df30be-kserve-88c5dcd79-qrhq2\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.592603 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.592524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-model-cache\") pod \"distinct-w0-df30be-kserve-88c5dcd79-qrhq2\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.592603 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.592570    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-home\") pod \"distinct-w0-df30be-kserve-88c5dcd79-qrhq2\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.592799 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.592606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxl5m\" (UniqueName: \"kubernetes.io/projected/dccfee92-8984-45fd-899b-7ab786c9525b-kube-api-access-rxl5m\") pod \"distinct-w0-df30be-kserve-88c5dcd79-qrhq2\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.592799 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.592664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccfee92-8984-45fd-899b-7ab786c9525b-tls-certs\") pod \"distinct-w0-df30be-kserve-88c5dcd79-qrhq2\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.592799 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.592691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-tmp-dir\") pod \"distinct-w0-df30be-kserve-88c5dcd79-qrhq2\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.592799 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.592722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-dshm\") pod \"distinct-w0-df30be-kserve-88c5dcd79-qrhq2\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.593082 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.592974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-model-cache\") pod \"distinct-w0-df30be-kserve-88c5dcd79-qrhq2\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.593082 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.593047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-home\") pod \"distinct-w0-df30be-kserve-88c5dcd79-qrhq2\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.593203 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.593174    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-tmp-dir\") pod \"distinct-w0-df30be-kserve-88c5dcd79-qrhq2\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.595082 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.595060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-dshm\") pod \"distinct-w0-df30be-kserve-88c5dcd79-qrhq2\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.595333 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.595317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccfee92-8984-45fd-899b-7ab786c9525b-tls-certs\") pod \"distinct-w0-df30be-kserve-88c5dcd79-qrhq2\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.600381 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.600357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxl5m\" (UniqueName: \"kubernetes.io/projected/dccfee92-8984-45fd-899b-7ab786c9525b-kube-api-access-rxl5m\") pod \"distinct-w0-df30be-kserve-88c5dcd79-qrhq2\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.650199 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.650168    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:46.933750 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:46.933720    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"]
Sep 09 15:42:46.935961 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:42:46.935929    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddccfee92_8984_45fd_899b_7ab786c9525b.slice/crio-c8516e24e9b975917665fdd303dc261060dd680ae3d013600ca289bab8d8e2fc WatchSource:0}: Error finding container c8516e24e9b975917665fdd303dc261060dd680ae3d013600ca289bab8d8e2fc: Status 404 returned error can't find the container with id c8516e24e9b975917665fdd303dc261060dd680ae3d013600ca289bab8d8e2fc
Sep 09 15:42:47.150334 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:47.150302    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2" event={"ID":"dccfee92-8984-45fd-899b-7ab786c9525b","Type":"ContainerStarted","Data":"586338c5d152b213cb89739e85a0237d6f08018f4ffd383f69116d2271242b57"}
Sep 09 15:42:47.150509 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:47.150341    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2" event={"ID":"dccfee92-8984-45fd-899b-7ab786c9525b","Type":"ContainerStarted","Data":"c8516e24e9b975917665fdd303dc261060dd680ae3d013600ca289bab8d8e2fc"}
Sep 09 15:42:47.166243 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:47.166191    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2" podStartSLOduration=1.166175901 podStartE2EDuration="1.166175901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:42:47.165883523 +0000 UTC m=+1356.807925518" watchObservedRunningTime="2026-09-09 15:42:47.166175901 +0000 UTC m=+1356.808217892"
Sep 09 15:42:53.410365 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:53.410325    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:56.373372 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:56.373329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:56.650434 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:56.650396    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:56.650642 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:56.650470    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:56.663620 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:56.663595    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:56.982853 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:56.982755    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"]
Sep 09 15:42:56.983059 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:56.983008    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz" podUID="1a9f3c95-2d2e-456d-a29e-a7eb2b9691da" containerName="maas-api" containerID="cri-o://d2d09a2483b0cda4f92ed604bece2a95631f2d62a98d7b90a6e7764d9089e4ac" gracePeriod=30
Sep 09 15:42:57.013841 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.013815    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:57.129579 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.129547    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:57.209628 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.209599    2578 generic.go:356] "Generic (PLEG): container finished" podID="1a9f3c95-2d2e-456d-a29e-a7eb2b9691da" containerID="d2d09a2483b0cda4f92ed604bece2a95631f2d62a98d7b90a6e7764d9089e4ac" exitCode=0
Sep 09 15:42:57.209833 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.209669    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz" event={"ID":"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da","Type":"ContainerDied","Data":"d2d09a2483b0cda4f92ed604bece2a95631f2d62a98d7b90a6e7764d9089e4ac"}
Sep 09 15:42:57.227802 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.227714    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:42:57.245859 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.245801    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:57.258481 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.258455    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:57.343520 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.343486    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:57.407693 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.407662    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-maas-api-tls\") pod \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\" (UID: \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\") "
Sep 09 15:42:57.408155 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.407775    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-metrics-certs\") pod \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\" (UID: \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\") "
Sep 09 15:42:57.408155 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.407800    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-openshift-service-ca\") pod \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\" (UID: \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\") "
Sep 09 15:42:57.408155 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.407891    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tzj7\" (UniqueName: \"kubernetes.io/projected/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-kube-api-access-9tzj7\") pod \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\" (UID: \"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da\") "
Sep 09 15:42:57.408330 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.408269    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1a9f3c95-2d2e-456d-a29e-a7eb2b9691da" (UID: "1a9f3c95-2d2e-456d-a29e-a7eb2b9691da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:42:57.410053 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.410028    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1a9f3c95-2d2e-456d-a29e-a7eb2b9691da" (UID: "1a9f3c95-2d2e-456d-a29e-a7eb2b9691da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:42:57.410600 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.410583    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1a9f3c95-2d2e-456d-a29e-a7eb2b9691da" (UID: "1a9f3c95-2d2e-456d-a29e-a7eb2b9691da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:42:57.410672 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.410611    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-kube-api-access-9tzj7" (OuterVolumeSpecName: "kube-api-access-9tzj7") pod "1a9f3c95-2d2e-456d-a29e-a7eb2b9691da" (UID: "1a9f3c95-2d2e-456d-a29e-a7eb2b9691da"). InnerVolumeSpecName "kube-api-access-9tzj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:42:57.509376 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.509290    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tzj7\" (UniqueName: \"kubernetes.io/projected/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-kube-api-access-9tzj7\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:42:57.509376 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.509321    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-maas-api-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:42:57.509376 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.509330    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:42:57.509376 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.509338    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da-openshift-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:42:57.678037 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:57.677996    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:42:58.216893 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:58.216864    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"
Sep 09 15:42:58.217105 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:58.216893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz" event={"ID":"1a9f3c95-2d2e-456d-a29e-a7eb2b9691da","Type":"ContainerDied","Data":"53d74948ff5c1adfe60febe05236d0400c6f8c667e90e8c980423ce8bb3b8c43"}
Sep 09 15:42:58.217105 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:58.216950    2578 scope.go:117] "RemoveContainer" containerID="d2d09a2483b0cda4f92ed604bece2a95631f2d62a98d7b90a6e7764d9089e4ac"
Sep 09 15:42:58.238574 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:58.238550    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"]
Sep 09 15:42:58.242571 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:58.242549    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-73d87788-f5848cbf4-8xmcz"]
Sep 09 15:42:58.868045 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:42:58.868014    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a9f3c95-2d2e-456d-a29e-a7eb2b9691da" path="/var/lib/kubelet/pods/1a9f3c95-2d2e-456d-a29e-a7eb2b9691da/volumes"
Sep 09 15:43:03.391241 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:03.391205    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:03.490750 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:03.490717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:03.590517 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:03.590476    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:03.751140 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:03.751053    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:03.972532 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:03.972504    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:04.074125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:04.074055    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:04.173088 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:04.173057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:04.274711 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:04.274678    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:04.351876 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:04.351791    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:04.563351 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:04.563319    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:10.299363 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:10.299329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:10.399280 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:10.399245    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:10.525133 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:10.525098    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:11.588696 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:11.588664    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:15.293383 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:15.293354    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:16.148700 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:16.148665    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:16.245768 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:16.245734    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:16.380374 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:16.380346    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:16.481299 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:16.481218    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:17.413901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:17.413870    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:17.739614 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:17.739542    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:17.839813 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:17.839782    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:18.515717 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:18.515689    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:20.533048 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:20.533016    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:20.625852 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:20.625819    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:21.680074 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:21.680043    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:21.780643 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:21.780611    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:21.989017 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:21.988920    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:22.262244 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:22.262162    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:22.363283 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:22.363253    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:45.677256 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:45.677223    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:45.996758 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:45.996683    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:46.370539 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:46.370455    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:47.016633 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:47.016601    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:47.115912 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:47.115872    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:47.285436 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:47.285350    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:47.780415 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:47.780384    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:48.118846 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:48.118762    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:48.750861 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:48.750817    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:51.639225 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:51.639187    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"]
Sep 09 15:43:51.639901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:51.639880    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a9f3c95-2d2e-456d-a29e-a7eb2b9691da" containerName="maas-api"
Sep 09 15:43:51.639949 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:51.639906    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a9f3c95-2d2e-456d-a29e-a7eb2b9691da" containerName="maas-api"
Sep 09 15:43:51.640095 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:51.640081    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a9f3c95-2d2e-456d-a29e-a7eb2b9691da" containerName="maas-api"
Sep 09 15:43:51.643917 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:51.643889    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:43:51.646280 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:51.646250    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-081ed3-serving-cert\""
Sep 09 15:43:51.654398 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:51.654350    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"]
Sep 09 15:43:51.764130 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:51.764094    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f\" (UID: \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:43:51.764300 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:51.764247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f\" (UID: \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:43:51.764300 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:51.764291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f\" (UID: \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:43:51.764391 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:51.764373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9s97\" (UniqueName: \"kubernetes.io/projected/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-kube-api-access-f9s97\") pod \"maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f\" (UID: \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:43:51.865532 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:51.865501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f\" (UID: \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:43:51.865719 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:51.865548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f\" (UID: \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:43:51.865719 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:51.865642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9s97\" (UniqueName: \"kubernetes.io/projected/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-kube-api-access-f9s97\") pod \"maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f\" (UID: \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:43:51.865719 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:43:51.865667    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-serving-cert: secret "maas-api-e2e-shared-b-w2-081ed3-serving-cert" not found
Sep 09 15:43:51.865719 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:51.865681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f\" (UID: \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:43:51.865943 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:43:51.865734    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-maas-api-tls podName:d19c9a9e-2070-4212-9f88-7e0dde8cfd44 nodeName:}" failed. No retries permitted until 2026-09-09 15:43:52.365711747 +0000 UTC m=+1422.007753731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-maas-api-tls") pod "maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f" (UID: "d19c9a9e-2070-4212-9f88-7e0dde8cfd44") : secret "maas-api-e2e-shared-b-w2-081ed3-serving-cert" not found
Sep 09 15:43:51.866381 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:51.866363    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f\" (UID: \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:43:51.868273 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:51.868251    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f\" (UID: \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:43:51.874763 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:51.874742    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9s97\" (UniqueName: \"kubernetes.io/projected/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-kube-api-access-f9s97\") pod \"maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f\" (UID: \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:43:52.084222 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:52.084145    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:52.182396 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:52.182363    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:52.255809 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:52.255753    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:43:52.371582 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:52.371503    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f\" (UID: \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:43:52.374170 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:52.374144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f\" (UID: \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:43:52.557418 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:52.557383    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:43:52.843520 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:52.843489    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"]
Sep 09 15:43:52.845290 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:43:52.845260    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19c9a9e_2070_4212_9f88_7e0dde8cfd44.slice/crio-6d34a00f67d71c27031039e0498f91b2ad2ad368428a1f95d4edab57865806c2 WatchSource:0}: Error finding container 6d34a00f67d71c27031039e0498f91b2ad2ad368428a1f95d4edab57865806c2: Status 404 returned error can't find the container with id 6d34a00f67d71c27031039e0498f91b2ad2ad368428a1f95d4edab57865806c2
Sep 09 15:43:52.846768 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:52.846744    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 15:43:53.523482 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:53.523449    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f" event={"ID":"d19c9a9e-2070-4212-9f88-7e0dde8cfd44","Type":"ContainerStarted","Data":"6d34a00f67d71c27031039e0498f91b2ad2ad368428a1f95d4edab57865806c2"}
Sep 09 15:43:57.548695 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:57.548653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f" event={"ID":"d19c9a9e-2070-4212-9f88-7e0dde8cfd44","Type":"ContainerStarted","Data":"bdf97ef0e5e59f7008c035ae695753a28b2b140db5ecb84841ad3603e7732940"}
Sep 09 15:43:57.549127 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:57.548802    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:43:57.568258 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:43:57.568213    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f" podStartSLOduration=2.912092807 podStartE2EDuration="6.568200109s" totalImagesPullingTime="3.656107302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:43:57.566799945 +0000 UTC m=+1427.208841933" watchObservedRunningTime="2026-09-09 15:43:57.568200109 +0000 UTC m=+1427.210242097"
Sep 09 15:44:03.560207 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:03.560129    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:44:10.924272 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:10.924231    2578 scope.go:117] "RemoveContainer" containerID="d634cf831c528c3855ae9575b2977eb7e9de5dbc608065ea1e8280efaf17ba1f"
Sep 09 15:44:15.845119 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:15.845089    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:44:19.657021 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.656971    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"]
Sep 09 15:44:19.662309 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.662290    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.665918 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.665900    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-df30be\"/\"distinct-2-w0-df30be-kserve-self-signed-certs\""
Sep 09 15:44:19.672178 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.672156    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"]
Sep 09 15:44:19.691275 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.691211    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:44:19.746006 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.745965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-dshm\") pod \"distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.746137 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.746016    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-tmp-dir\") pod \"distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.746137 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.746076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpf6w\" (UniqueName: \"kubernetes.io/projected/6f69e13f-5e3a-4cac-a449-72458cb37ccc-kube-api-access-jpf6w\") pod \"distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.746137 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.746128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-home\") pod \"distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.746257 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.746142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-model-cache\") pod \"distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.746293 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.746249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f69e13f-5e3a-4cac-a449-72458cb37ccc-tls-certs\") pod \"distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.847142 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.847090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpf6w\" (UniqueName: \"kubernetes.io/projected/6f69e13f-5e3a-4cac-a449-72458cb37ccc-kube-api-access-jpf6w\") pod \"distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.847346 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.847187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-home\") pod \"distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.847346 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.847210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-model-cache\") pod \"distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.847346 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.847271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f69e13f-5e3a-4cac-a449-72458cb37ccc-tls-certs\") pod \"distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.847542 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.847369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-dshm\") pod \"distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.847542 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.847401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-tmp-dir\") pod \"distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.847673 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.847649    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-home\") pod \"distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.847742 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.847658    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-model-cache\") pod \"distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.847800 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.847761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-tmp-dir\") pod \"distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.849848 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.849825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-dshm\") pod \"distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.850041 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.850024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f69e13f-5e3a-4cac-a449-72458cb37ccc-tls-certs\") pod \"distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.855884 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.855862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpf6w\" (UniqueName: \"kubernetes.io/projected/6f69e13f-5e3a-4cac-a449-72458cb37ccc-kube-api-access-jpf6w\") pod \"distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") " pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:19.971661 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:19.971562    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:20.244167 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:20.244143    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"]
Sep 09 15:44:20.246066 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:44:20.246038    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f69e13f_5e3a_4cac_a449_72458cb37ccc.slice/crio-edfc55daf686fc72ad55011bd626006189f1551f0d5c62e6e1f684f46dd40e5d WatchSource:0}: Error finding container edfc55daf686fc72ad55011bd626006189f1551f0d5c62e6e1f684f46dd40e5d: Status 404 returned error can't find the container with id edfc55daf686fc72ad55011bd626006189f1551f0d5c62e6e1f684f46dd40e5d
Sep 09 15:44:20.686080 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:20.686047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg" event={"ID":"6f69e13f-5e3a-4cac-a449-72458cb37ccc","Type":"ContainerStarted","Data":"aa3fcc36cd98b1e1c6d36463eb4ed0214c1539864b4823eb6896c151db028d8c"}
Sep 09 15:44:20.686080 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:20.686089    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg" event={"ID":"6f69e13f-5e3a-4cac-a449-72458cb37ccc","Type":"ContainerStarted","Data":"edfc55daf686fc72ad55011bd626006189f1551f0d5c62e6e1f684f46dd40e5d"}
Sep 09 15:44:20.704858 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:20.704816    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg" podStartSLOduration=1.70480027 podStartE2EDuration="1.70480027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:44:20.704475368 +0000 UTC m=+1450.346517358" watchObservedRunningTime="2026-09-09 15:44:20.70480027 +0000 UTC m=+1450.346842256"
Sep 09 15:44:29.899799 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:29.899759    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:44:29.971997 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:29.971936    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:29.972190 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:29.972007    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:29.984503 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:29.984475    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:30.233248 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:30.233156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:44:30.623206 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:30.623123    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:44:30.753315 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:30.753285    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:44:33.082413 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:33.082378    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:44:33.199255 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:33.199223    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:44:34.358438 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.358409    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"]
Sep 09 15:44:34.362958 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.362936    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"
Sep 09 15:44:34.365341 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.365322    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-aeb0f8ca-serving-cert\""
Sep 09 15:44:34.375232 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.375202    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"]
Sep 09 15:44:34.397480 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.397452    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:44:34.497992 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.497957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:44:34.502197 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.502175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4cff\" (UniqueName: \"kubernetes.io/projected/eaa99199-675b-4029-8f81-a0896035d284-kube-api-access-m4cff\") pod \"maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k\" (UID: \"eaa99199-675b-4029-8f81-a0896035d284\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"
Sep 09 15:44:34.502338 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.502313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa99199-675b-4029-8f81-a0896035d284-maas-api-tls\") pod \"maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k\" (UID: \"eaa99199-675b-4029-8f81-a0896035d284\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"
Sep 09 15:44:34.502410 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.502364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaa99199-675b-4029-8f81-a0896035d284-metrics-certs\") pod \"maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k\" (UID: \"eaa99199-675b-4029-8f81-a0896035d284\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"
Sep 09 15:44:34.502410 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.502397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaa99199-675b-4029-8f81-a0896035d284-openshift-service-ca\") pod \"maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k\" (UID: \"eaa99199-675b-4029-8f81-a0896035d284\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"
Sep 09 15:44:34.604111 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.604074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4cff\" (UniqueName: \"kubernetes.io/projected/eaa99199-675b-4029-8f81-a0896035d284-kube-api-access-m4cff\") pod \"maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k\" (UID: \"eaa99199-675b-4029-8f81-a0896035d284\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"
Sep 09 15:44:34.604307 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.604166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa99199-675b-4029-8f81-a0896035d284-maas-api-tls\") pod \"maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k\" (UID: \"eaa99199-675b-4029-8f81-a0896035d284\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"
Sep 09 15:44:34.604307 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.604183    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaa99199-675b-4029-8f81-a0896035d284-metrics-certs\") pod \"maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k\" (UID: \"eaa99199-675b-4029-8f81-a0896035d284\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"
Sep 09 15:44:34.604307 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.604202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaa99199-675b-4029-8f81-a0896035d284-openshift-service-ca\") pod \"maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k\" (UID: \"eaa99199-675b-4029-8f81-a0896035d284\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"
Sep 09 15:44:34.604943 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.604867    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaa99199-675b-4029-8f81-a0896035d284-openshift-service-ca\") pod \"maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k\" (UID: \"eaa99199-675b-4029-8f81-a0896035d284\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"
Sep 09 15:44:34.606722 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.606696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaa99199-675b-4029-8f81-a0896035d284-metrics-certs\") pod \"maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k\" (UID: \"eaa99199-675b-4029-8f81-a0896035d284\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"
Sep 09 15:44:34.606795 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.606731    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa99199-675b-4029-8f81-a0896035d284-maas-api-tls\") pod \"maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k\" (UID: \"eaa99199-675b-4029-8f81-a0896035d284\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"
Sep 09 15:44:34.614740 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.614689    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4cff\" (UniqueName: \"kubernetes.io/projected/eaa99199-675b-4029-8f81-a0896035d284-kube-api-access-m4cff\") pod \"maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k\" (UID: \"eaa99199-675b-4029-8f81-a0896035d284\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"
Sep 09 15:44:34.634483 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.634458    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:44:34.676988 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.676957    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"
Sep 09 15:44:34.958452 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.958413    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"]
Sep 09 15:44:34.959837 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:44:34.959810    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa99199_675b_4029_8f81_a0896035d284.slice/crio-d781cad1365f39dbb70ea73991755deb38c5c4a165d5319659b9740062914e7d WatchSource:0}: Error finding container d781cad1365f39dbb70ea73991755deb38c5c4a165d5319659b9740062914e7d: Status 404 returned error can't find the container with id d781cad1365f39dbb70ea73991755deb38c5c4a165d5319659b9740062914e7d
Sep 09 15:44:34.984066 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:34.984043    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:44:35.138716 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:35.138687    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:44:35.771422 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:35.771340    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k" event={"ID":"eaa99199-675b-4029-8f81-a0896035d284","Type":"ContainerStarted","Data":"8253f80ccf45b8a1717dcf8f57a25f0e65a76d31a21ebee41eaaa64c92a562ba"}
Sep 09 15:44:35.771422 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:35.771393    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k" event={"ID":"eaa99199-675b-4029-8f81-a0896035d284","Type":"ContainerStarted","Data":"d781cad1365f39dbb70ea73991755deb38c5c4a165d5319659b9740062914e7d"}
Sep 09 15:44:35.771807 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:35.771461    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"
Sep 09 15:44:35.789781 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:35.789723    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k" podStartSLOduration=1.292899442 podStartE2EDuration="1.789706385s" totalImagesPullingTime="496.806943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:44:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:44:35.787743771 +0000 UTC m=+1465.429785807" watchObservedRunningTime="2026-09-09 15:44:35.789706385 +0000 UTC m=+1465.431748374"
Sep 09 15:44:41.783013 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:41.782970    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"
Sep 09 15:44:52.984171 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:52.984134    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:44:55.367781 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:55.367745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:44:58.062357 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:44:58.062299    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:02.933384 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:02.933321    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:03.563186 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:03.563158    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:03.776114 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:03.776083    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:04.468076 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:04.468048    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:04.559041 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:04.559006    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:04.644390 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:04.644357    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:04.749479 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:04.749400    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:05.069915 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:05.069828    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:10.069897 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:10.069858    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"]
Sep 09 15:45:10.159837 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:10.159809    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:15.179471 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:15.179443    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 15:45:15.182268 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:15.182244    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 15:45:15.186973 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:15.186954    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:45:15.189552 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:15.189535    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:45:25.090434 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.090381    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll" podUID="0064268b-5817-4b48-8fea-ed07df45693f" containerName="main" containerID="cri-o://a508efaa24a68cbd93a90247ffd3da17920b5880c3b02e87940ff50837e7141b" gracePeriod=45
Sep 09 15:45:25.341091 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.341033    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:45:25.430588 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.430557    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-tmp-dir\") pod \"0064268b-5817-4b48-8fea-ed07df45693f\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") "
Sep 09 15:45:25.430744 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.430627    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-home\") pod \"0064268b-5817-4b48-8fea-ed07df45693f\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") "
Sep 09 15:45:25.430744 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.430661    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-dshm\") pod \"0064268b-5817-4b48-8fea-ed07df45693f\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") "
Sep 09 15:45:25.430744 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.430694    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0064268b-5817-4b48-8fea-ed07df45693f-tls-certs\") pod \"0064268b-5817-4b48-8fea-ed07df45693f\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") "
Sep 09 15:45:25.430744 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.430722    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-model-cache\") pod \"0064268b-5817-4b48-8fea-ed07df45693f\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") "
Sep 09 15:45:25.430939 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.430749    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4gsx\" (UniqueName: \"kubernetes.io/projected/0064268b-5817-4b48-8fea-ed07df45693f-kube-api-access-s4gsx\") pod \"0064268b-5817-4b48-8fea-ed07df45693f\" (UID: \"0064268b-5817-4b48-8fea-ed07df45693f\") "
Sep 09 15:45:25.430939 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.430874    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-home" (OuterVolumeSpecName: "home") pod "0064268b-5817-4b48-8fea-ed07df45693f" (UID: "0064268b-5817-4b48-8fea-ed07df45693f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:45:25.430939 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.430883    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0064268b-5817-4b48-8fea-ed07df45693f" (UID: "0064268b-5817-4b48-8fea-ed07df45693f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:45:25.431204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.431039    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-model-cache" (OuterVolumeSpecName: "model-cache") pod "0064268b-5817-4b48-8fea-ed07df45693f" (UID: "0064268b-5817-4b48-8fea-ed07df45693f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:45:25.431269 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.431244    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-tmp-dir\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:25.431304 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.431275    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-home\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:25.431304 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.431288    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-model-cache\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:25.433114 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.433079    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0064268b-5817-4b48-8fea-ed07df45693f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0064268b-5817-4b48-8fea-ed07df45693f" (UID: "0064268b-5817-4b48-8fea-ed07df45693f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:45:25.433221 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.433114    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-dshm" (OuterVolumeSpecName: "dshm") pod "0064268b-5817-4b48-8fea-ed07df45693f" (UID: "0064268b-5817-4b48-8fea-ed07df45693f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:45:25.433221 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.433114    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0064268b-5817-4b48-8fea-ed07df45693f-kube-api-access-s4gsx" (OuterVolumeSpecName: "kube-api-access-s4gsx") pod "0064268b-5817-4b48-8fea-ed07df45693f" (UID: "0064268b-5817-4b48-8fea-ed07df45693f"). InnerVolumeSpecName "kube-api-access-s4gsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:45:25.532540 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.532506    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0064268b-5817-4b48-8fea-ed07df45693f-dshm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:25.532540 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.532533    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0064268b-5817-4b48-8fea-ed07df45693f-tls-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:25.532540 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:25.532545    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4gsx\" (UniqueName: \"kubernetes.io/projected/0064268b-5817-4b48-8fea-ed07df45693f-kube-api-access-s4gsx\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:26.057704 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:26.057670    2578 generic.go:356] "Generic (PLEG): container finished" podID="0064268b-5817-4b48-8fea-ed07df45693f" containerID="a508efaa24a68cbd93a90247ffd3da17920b5880c3b02e87940ff50837e7141b" exitCode=0
Sep 09 15:45:26.057902 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:26.057719    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"
Sep 09 15:45:26.057902 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:26.057748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll" event={"ID":"0064268b-5817-4b48-8fea-ed07df45693f","Type":"ContainerDied","Data":"a508efaa24a68cbd93a90247ffd3da17920b5880c3b02e87940ff50837e7141b"}
Sep 09 15:45:26.057902 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:26.057788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll" event={"ID":"0064268b-5817-4b48-8fea-ed07df45693f","Type":"ContainerDied","Data":"c20a592380449eafb7465f48995b3cdd7f83aec7748b81ad50c4291bff163afa"}
Sep 09 15:45:26.057902 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:26.057805    2578 scope.go:117] "RemoveContainer" containerID="a508efaa24a68cbd93a90247ffd3da17920b5880c3b02e87940ff50837e7141b"
Sep 09 15:45:26.067937 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:26.067918    2578 scope.go:117] "RemoveContainer" containerID="a508efaa24a68cbd93a90247ffd3da17920b5880c3b02e87940ff50837e7141b"
Sep 09 15:45:26.068256 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:45:26.068239    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a508efaa24a68cbd93a90247ffd3da17920b5880c3b02e87940ff50837e7141b\": container with ID starting with a508efaa24a68cbd93a90247ffd3da17920b5880c3b02e87940ff50837e7141b not found: ID does not exist" containerID="a508efaa24a68cbd93a90247ffd3da17920b5880c3b02e87940ff50837e7141b"
Sep 09 15:45:26.068318 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:26.068264    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a508efaa24a68cbd93a90247ffd3da17920b5880c3b02e87940ff50837e7141b"} err="failed to get container status \"a508efaa24a68cbd93a90247ffd3da17920b5880c3b02e87940ff50837e7141b\": rpc error: code = NotFound desc = could not find container \"a508efaa24a68cbd93a90247ffd3da17920b5880c3b02e87940ff50837e7141b\": container with ID starting with a508efaa24a68cbd93a90247ffd3da17920b5880c3b02e87940ff50837e7141b not found: ID does not exist"
Sep 09 15:45:26.079605 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:26.079583    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"]
Sep 09 15:45:26.082678 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:26.082657    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-57b579/test-model-57b579-kserve-7f589c9f9f-w4lll"]
Sep 09 15:45:26.867771 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:26.867730    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0064268b-5817-4b48-8fea-ed07df45693f" path="/var/lib/kubelet/pods/0064268b-5817-4b48-8fea-ed07df45693f/volumes"
Sep 09 15:45:30.048337 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:30.048308    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:33.150678 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:33.150603    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:46.851615 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:46.851585    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:47.123310 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:47.123226    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"]
Sep 09 15:45:47.123502 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:47.123475    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk" podUID="5bb883ca-8542-4127-b9be-2ba9571c8c93" containerName="istio-proxy" containerID="cri-o://a851a9861ce1ec900611ac7ececfed46cf706103849e8fb8666957ec3ea69eba" gracePeriod=30
Sep 09 15:45:47.146741 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:47.146716    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:47.659670 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:47.659640    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:49.056908 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:49.056877    2578 patch_prober.go:28] interesting pod/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 15:45:49.057289 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:49.056944    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk" podUID="5bb883ca-8542-4127-b9be-2ba9571c8c93" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 15:45:52.371478 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.371455    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:45:52.506084 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.506006    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-credential-socket\") pod \"5bb883ca-8542-4127-b9be-2ba9571c8c93\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") "
Sep 09 15:45:52.506084 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.506039    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-workload-certs\") pod \"5bb883ca-8542-4127-b9be-2ba9571c8c93\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") "
Sep 09 15:45:52.506315 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.506088    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-workload-socket\") pod \"5bb883ca-8542-4127-b9be-2ba9571c8c93\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") "
Sep 09 15:45:52.506315 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.506111    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb883ca-8542-4127-b9be-2ba9571c8c93-istiod-ca-cert\") pod \"5bb883ca-8542-4127-b9be-2ba9571c8c93\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") "
Sep 09 15:45:52.506315 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.506157    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-token\") pod \"5bb883ca-8542-4127-b9be-2ba9571c8c93\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") "
Sep 09 15:45:52.506315 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.506196    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-podinfo\") pod \"5bb883ca-8542-4127-b9be-2ba9571c8c93\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") "
Sep 09 15:45:52.506315 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.506229    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-data\") pod \"5bb883ca-8542-4127-b9be-2ba9571c8c93\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") "
Sep 09 15:45:52.506315 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.506259    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-envoy\") pod \"5bb883ca-8542-4127-b9be-2ba9571c8c93\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") "
Sep 09 15:45:52.506315 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.506282    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5bb883ca-8542-4127-b9be-2ba9571c8c93" (UID: "5bb883ca-8542-4127-b9be-2ba9571c8c93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:45:52.506678 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.506324    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsvpr\" (UniqueName: \"kubernetes.io/projected/5bb883ca-8542-4127-b9be-2ba9571c8c93-kube-api-access-jsvpr\") pod \"5bb883ca-8542-4127-b9be-2ba9571c8c93\" (UID: \"5bb883ca-8542-4127-b9be-2ba9571c8c93\") "
Sep 09 15:45:52.506678 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.506363    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5bb883ca-8542-4127-b9be-2ba9571c8c93" (UID: "5bb883ca-8542-4127-b9be-2ba9571c8c93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:45:52.506678 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.506602    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5bb883ca-8542-4127-b9be-2ba9571c8c93" (UID: "5bb883ca-8542-4127-b9be-2ba9571c8c93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:45:52.506678 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.506655    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb883ca-8542-4127-b9be-2ba9571c8c93-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5bb883ca-8542-4127-b9be-2ba9571c8c93" (UID: "5bb883ca-8542-4127-b9be-2ba9571c8c93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:45:52.506876 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.506803    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb883ca-8542-4127-b9be-2ba9571c8c93-istiod-ca-cert\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:52.506876 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.506822    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-credential-socket\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:52.506876 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.506836    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-workload-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:52.506876 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.506859    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-workload-socket\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:52.507059 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.506921    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-data" (OuterVolumeSpecName: "istio-data") pod "5bb883ca-8542-4127-b9be-2ba9571c8c93" (UID: "5bb883ca-8542-4127-b9be-2ba9571c8c93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:45:52.508727 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.508704    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb883ca-8542-4127-b9be-2ba9571c8c93-kube-api-access-jsvpr" (OuterVolumeSpecName: "kube-api-access-jsvpr") pod "5bb883ca-8542-4127-b9be-2ba9571c8c93" (UID: "5bb883ca-8542-4127-b9be-2ba9571c8c93"). InnerVolumeSpecName "kube-api-access-jsvpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:45:52.508859 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.508850    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5bb883ca-8542-4127-b9be-2ba9571c8c93" (UID: "5bb883ca-8542-4127-b9be-2ba9571c8c93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:45:52.509173 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.509149    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-token" (OuterVolumeSpecName: "istio-token") pod "5bb883ca-8542-4127-b9be-2ba9571c8c93" (UID: "5bb883ca-8542-4127-b9be-2ba9571c8c93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:45:52.509173 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.509152    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5bb883ca-8542-4127-b9be-2ba9571c8c93" (UID: "5bb883ca-8542-4127-b9be-2ba9571c8c93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 15:45:52.607911 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.607873    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsvpr\" (UniqueName: \"kubernetes.io/projected/5bb883ca-8542-4127-b9be-2ba9571c8c93-kube-api-access-jsvpr\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:52.607911 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.607904    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-token\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:52.607911 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.607913    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-podinfo\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:52.608156 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.607920    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-data\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:52.608156 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:52.607929    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb883ca-8542-4127-b9be-2ba9571c8c93-istio-envoy\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:53.210880 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:53.210837    2578 generic.go:356] "Generic (PLEG): container finished" podID="5bb883ca-8542-4127-b9be-2ba9571c8c93" containerID="a851a9861ce1ec900611ac7ececfed46cf706103849e8fb8666957ec3ea69eba" exitCode=0
Sep 09 15:45:53.211076 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:53.210897    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"
Sep 09 15:45:53.211076 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:53.210927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk" event={"ID":"5bb883ca-8542-4127-b9be-2ba9571c8c93","Type":"ContainerDied","Data":"a851a9861ce1ec900611ac7ececfed46cf706103849e8fb8666957ec3ea69eba"}
Sep 09 15:45:53.211076 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:53.211008    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk" event={"ID":"5bb883ca-8542-4127-b9be-2ba9571c8c93","Type":"ContainerDied","Data":"0ea61334c50bf29f87893ed979816e109d5e5fd3389eb4bcf4572da7a0002c09"}
Sep 09 15:45:53.211076 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:53.211016    2578 scope.go:117] "RemoveContainer" containerID="a851a9861ce1ec900611ac7ececfed46cf706103849e8fb8666957ec3ea69eba"
Sep 09 15:45:53.221181 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:53.221165    2578 scope.go:117] "RemoveContainer" containerID="07613ebb5db0edf16932f290c59c23c0630f9c62e3a87b551fb0616e5f68637c"
Sep 09 15:45:53.236001 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:53.235949    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"]
Sep 09 15:45:53.239855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:53.239716    2578 scope.go:117] "RemoveContainer" containerID="a851a9861ce1ec900611ac7ececfed46cf706103849e8fb8666957ec3ea69eba"
Sep 09 15:45:53.240295 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:45:53.240273    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a851a9861ce1ec900611ac7ececfed46cf706103849e8fb8666957ec3ea69eba\": container with ID starting with a851a9861ce1ec900611ac7ececfed46cf706103849e8fb8666957ec3ea69eba not found: ID does not exist" containerID="a851a9861ce1ec900611ac7ececfed46cf706103849e8fb8666957ec3ea69eba"
Sep 09 15:45:53.240397 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:53.240305    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a851a9861ce1ec900611ac7ececfed46cf706103849e8fb8666957ec3ea69eba"} err="failed to get container status \"a851a9861ce1ec900611ac7ececfed46cf706103849e8fb8666957ec3ea69eba\": rpc error: code = NotFound desc = could not find container \"a851a9861ce1ec900611ac7ececfed46cf706103849e8fb8666957ec3ea69eba\": container with ID starting with a851a9861ce1ec900611ac7ececfed46cf706103849e8fb8666957ec3ea69eba not found: ID does not exist"
Sep 09 15:45:53.240397 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:53.240331    2578 scope.go:117] "RemoveContainer" containerID="07613ebb5db0edf16932f290c59c23c0630f9c62e3a87b551fb0616e5f68637c"
Sep 09 15:45:53.240599 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:45:53.240583    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07613ebb5db0edf16932f290c59c23c0630f9c62e3a87b551fb0616e5f68637c\": container with ID starting with 07613ebb5db0edf16932f290c59c23c0630f9c62e3a87b551fb0616e5f68637c not found: ID does not exist" containerID="07613ebb5db0edf16932f290c59c23c0630f9c62e3a87b551fb0616e5f68637c"
Sep 09 15:45:53.240652 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:53.240606    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07613ebb5db0edf16932f290c59c23c0630f9c62e3a87b551fb0616e5f68637c"} err="failed to get container status \"07613ebb5db0edf16932f290c59c23c0630f9c62e3a87b551fb0616e5f68637c\": rpc error: code = NotFound desc = could not find container \"07613ebb5db0edf16932f290c59c23c0630f9c62e3a87b551fb0616e5f68637c\": container with ID starting with 07613ebb5db0edf16932f290c59c23c0630f9c62e3a87b551fb0616e5f68637c not found: ID does not exist"
Sep 09 15:45:53.240770 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:53.240747    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-57b579-openshift-default-758589b7fb-2trwk"]
Sep 09 15:45:53.954940 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:53.954903    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"]
Sep 09 15:45:53.974355 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:53.974326    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:54.867177 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:54.867149    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bb883ca-8542-4127-b9be-2ba9571c8c93" path="/var/lib/kubelet/pods/5bb883ca-8542-4127-b9be-2ba9571c8c93/volumes"
Sep 09 15:45:57.956395 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:57.956363    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"]
Sep 09 15:45:57.957009 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:57.956569    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p" podUID="67b70461-1b17-4739-8660-51f9b4f2a2d9" containerName="maas-api" containerID="cri-o://e43a351f12f37f16182b7ee4b3015f06a0f9a91435a09b3f726d0c71d2d20365" gracePeriod=30
Sep 09 15:45:58.007907 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.007875    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:58.107675 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.107644    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:58.175167 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.175139    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:58.224676 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.224648    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:45:58.231164 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.231145    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:58.237686 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.237659    2578 generic.go:356] "Generic (PLEG): container finished" podID="67b70461-1b17-4739-8660-51f9b4f2a2d9" containerID="e43a351f12f37f16182b7ee4b3015f06a0f9a91435a09b3f726d0c71d2d20365" exitCode=0
Sep 09 15:45:58.237784 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.237699    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"
Sep 09 15:45:58.237784 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.237732    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p" event={"ID":"67b70461-1b17-4739-8660-51f9b4f2a2d9","Type":"ContainerDied","Data":"e43a351f12f37f16182b7ee4b3015f06a0f9a91435a09b3f726d0c71d2d20365"}
Sep 09 15:45:58.237784 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.237777    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p" event={"ID":"67b70461-1b17-4739-8660-51f9b4f2a2d9","Type":"ContainerDied","Data":"5120aba5f23e2427eb48e15f2e03952f3da473bbd759f70df7e7faab043a3061"}
Sep 09 15:45:58.237904 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.237801    2578 scope.go:117] "RemoveContainer" containerID="e43a351f12f37f16182b7ee4b3015f06a0f9a91435a09b3f726d0c71d2d20365"
Sep 09 15:45:58.249339 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.249319    2578 scope.go:117] "RemoveContainer" containerID="e43a351f12f37f16182b7ee4b3015f06a0f9a91435a09b3f726d0c71d2d20365"
Sep 09 15:45:58.249838 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:45:58.249724    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e43a351f12f37f16182b7ee4b3015f06a0f9a91435a09b3f726d0c71d2d20365\": container with ID starting with e43a351f12f37f16182b7ee4b3015f06a0f9a91435a09b3f726d0c71d2d20365 not found: ID does not exist" containerID="e43a351f12f37f16182b7ee4b3015f06a0f9a91435a09b3f726d0c71d2d20365"
Sep 09 15:45:58.249838 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.249769    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e43a351f12f37f16182b7ee4b3015f06a0f9a91435a09b3f726d0c71d2d20365"} err="failed to get container status \"e43a351f12f37f16182b7ee4b3015f06a0f9a91435a09b3f726d0c71d2d20365\": rpc error: code = NotFound desc = could not find container \"e43a351f12f37f16182b7ee4b3015f06a0f9a91435a09b3f726d0c71d2d20365\": container with ID starting with e43a351f12f37f16182b7ee4b3015f06a0f9a91435a09b3f726d0c71d2d20365 not found: ID does not exist"
Sep 09 15:45:58.262044 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.262019    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b70461-1b17-4739-8660-51f9b4f2a2d9-openshift-service-ca\") pod \"67b70461-1b17-4739-8660-51f9b4f2a2d9\" (UID: \"67b70461-1b17-4739-8660-51f9b4f2a2d9\") "
Sep 09 15:45:58.262200 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.262182    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67b70461-1b17-4739-8660-51f9b4f2a2d9-maas-api-tls\") pod \"67b70461-1b17-4739-8660-51f9b4f2a2d9\" (UID: \"67b70461-1b17-4739-8660-51f9b4f2a2d9\") "
Sep 09 15:45:58.262286 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.262234    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4nls\" (UniqueName: \"kubernetes.io/projected/67b70461-1b17-4739-8660-51f9b4f2a2d9-kube-api-access-d4nls\") pod \"67b70461-1b17-4739-8660-51f9b4f2a2d9\" (UID: \"67b70461-1b17-4739-8660-51f9b4f2a2d9\") "
Sep 09 15:45:58.262286 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.262273    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b70461-1b17-4739-8660-51f9b4f2a2d9-metrics-certs\") pod \"67b70461-1b17-4739-8660-51f9b4f2a2d9\" (UID: \"67b70461-1b17-4739-8660-51f9b4f2a2d9\") "
Sep 09 15:45:58.262468 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.262441    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b70461-1b17-4739-8660-51f9b4f2a2d9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67b70461-1b17-4739-8660-51f9b4f2a2d9" (UID: "67b70461-1b17-4739-8660-51f9b4f2a2d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:45:58.262721 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.262702    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b70461-1b17-4739-8660-51f9b4f2a2d9-openshift-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:58.264479 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.264457    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67b70461-1b17-4739-8660-51f9b4f2a2d9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67b70461-1b17-4739-8660-51f9b4f2a2d9" (UID: "67b70461-1b17-4739-8660-51f9b4f2a2d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:45:58.264673 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.264649    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67b70461-1b17-4739-8660-51f9b4f2a2d9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "67b70461-1b17-4739-8660-51f9b4f2a2d9" (UID: "67b70461-1b17-4739-8660-51f9b4f2a2d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:45:58.265094 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.265078    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b70461-1b17-4739-8660-51f9b4f2a2d9-kube-api-access-d4nls" (OuterVolumeSpecName: "kube-api-access-d4nls") pod "67b70461-1b17-4739-8660-51f9b4f2a2d9" (UID: "67b70461-1b17-4739-8660-51f9b4f2a2d9"). InnerVolumeSpecName "kube-api-access-d4nls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:45:58.364132 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.364094    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67b70461-1b17-4739-8660-51f9b4f2a2d9-maas-api-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:58.364132 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.364127    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4nls\" (UniqueName: \"kubernetes.io/projected/67b70461-1b17-4739-8660-51f9b4f2a2d9-kube-api-access-d4nls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:58.364333 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.364144    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b70461-1b17-4739-8660-51f9b4f2a2d9-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:45:58.561639 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.561612    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"]
Sep 09 15:45:58.567400 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.567376    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-267962-58d68778cf-p5x4p"]
Sep 09 15:45:58.592898 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.592877    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:45:58.867578 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:45:58.867498    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67b70461-1b17-4739-8660-51f9b4f2a2d9" path="/var/lib/kubelet/pods/67b70461-1b17-4739-8660-51f9b4f2a2d9/volumes"
Sep 09 15:46:01.855852 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.855816    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"]
Sep 09 15:46:01.856372 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.856356    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0064268b-5817-4b48-8fea-ed07df45693f" containerName="main"
Sep 09 15:46:01.856422 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.856374    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0064268b-5817-4b48-8fea-ed07df45693f" containerName="main"
Sep 09 15:46:01.856422 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.856399    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b70461-1b17-4739-8660-51f9b4f2a2d9" containerName="maas-api"
Sep 09 15:46:01.856422 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.856406    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b70461-1b17-4739-8660-51f9b4f2a2d9" containerName="maas-api"
Sep 09 15:46:01.856422 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.856415    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb883ca-8542-4127-b9be-2ba9571c8c93" containerName="istio-proxy"
Sep 09 15:46:01.856422 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.856421    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb883ca-8542-4127-b9be-2ba9571c8c93" containerName="istio-proxy"
Sep 09 15:46:01.856575 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.856427    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb883ca-8542-4127-b9be-2ba9571c8c93" containerName="istio-proxy"
Sep 09 15:46:01.856575 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.856432    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb883ca-8542-4127-b9be-2ba9571c8c93" containerName="istio-proxy"
Sep 09 15:46:01.856575 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.856502    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="67b70461-1b17-4739-8660-51f9b4f2a2d9" containerName="maas-api"
Sep 09 15:46:01.856575 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.856511    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb883ca-8542-4127-b9be-2ba9571c8c93" containerName="istio-proxy"
Sep 09 15:46:01.856575 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.856517    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0064268b-5817-4b48-8fea-ed07df45693f" containerName="main"
Sep 09 15:46:01.856575 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.856524    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb883ca-8542-4127-b9be-2ba9571c8c93" containerName="istio-proxy"
Sep 09 15:46:01.861200 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.861169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:01.864402 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.864381    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-aeb0f8ca\"/\"default-dockercfg-v95ch\""
Sep 09 15:46:01.864541 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.864381    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-aeb0f8ca\"/\"openshift-service-ca.crt\""
Sep 09 15:46:01.864541 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.864381    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-aeb0f8ca\"/\"kube-root-ca.crt\""
Sep 09 15:46:01.864541 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.864424    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-aeb0f8ca\"/\"e2e-lifecycle-model-aeb0f8ca-kserve-self-signed-certs\""
Sep 09 15:46:01.868445 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.868419    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"]
Sep 09 15:46:01.897799 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.897765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-dshm\") pod \"e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") " pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:01.897958 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.897808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-home\") pod \"e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") " pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:01.897958 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.897905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-tmp-dir\") pod \"e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") " pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:01.898098 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.897958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92791d59-620c-4aef-a8eb-65cd4db02b34-tls-certs\") pod \"e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") " pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:01.898098 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.898063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvpw2\" (UniqueName: \"kubernetes.io/projected/92791d59-620c-4aef-a8eb-65cd4db02b34-kube-api-access-mvpw2\") pod \"e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") " pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:01.898223 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.898101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-model-cache\") pod \"e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") " pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:01.962160 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.962131    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:01.999627 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.999599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvpw2\" (UniqueName: \"kubernetes.io/projected/92791d59-620c-4aef-a8eb-65cd4db02b34-kube-api-access-mvpw2\") pod \"e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") " pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:01.999729 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.999640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-model-cache\") pod \"e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") " pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:01.999729 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.999682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-dshm\") pod \"e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") " pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:01.999729 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.999712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-home\") pod \"e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") " pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:01.999850 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.999760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-tmp-dir\") pod \"e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") " pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:01.999850 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:01.999806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92791d59-620c-4aef-a8eb-65cd4db02b34-tls-certs\") pod \"e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") " pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:02.000140 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:02.000117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-home\") pod \"e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") " pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:02.000232 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:02.000205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-model-cache\") pod \"e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") " pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:02.000295 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:02.000275    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-tmp-dir\") pod \"e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") " pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:02.002270 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:02.002249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-dshm\") pod \"e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") " pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:02.002399 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:02.002381    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92791d59-620c-4aef-a8eb-65cd4db02b34-tls-certs\") pod \"e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") " pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:02.007229 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:02.007199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvpw2\" (UniqueName: \"kubernetes.io/projected/92791d59-620c-4aef-a8eb-65cd4db02b34-kube-api-access-mvpw2\") pod \"e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") " pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:02.173147 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:02.173112    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:02.445642 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:02.445617    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"]
Sep 09 15:46:02.447538 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:46:02.447510    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92791d59_620c_4aef_a8eb_65cd4db02b34.slice/crio-27d3d9c0ef02bec288efd225945a54170d464fc5f85c0e901c48d3509de51b55 WatchSource:0}: Error finding container 27d3d9c0ef02bec288efd225945a54170d464fc5f85c0e901c48d3509de51b55: Status 404 returned error can't find the container with id 27d3d9c0ef02bec288efd225945a54170d464fc5f85c0e901c48d3509de51b55
Sep 09 15:46:03.266770 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:03.266734    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g" event={"ID":"92791d59-620c-4aef-a8eb-65cd4db02b34","Type":"ContainerStarted","Data":"8e828d3cde7407403af5c56290b8def970d43ff9533935dc245998ee7d9b3443"}
Sep 09 15:46:03.267182 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:03.266783    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g" event={"ID":"92791d59-620c-4aef-a8eb-65cd4db02b34","Type":"ContainerStarted","Data":"27d3d9c0ef02bec288efd225945a54170d464fc5f85c0e901c48d3509de51b55"}
Sep 09 15:46:03.282136 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:03.282087    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g" podStartSLOduration=2.282072808 podStartE2EDuration="2.282072808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:46:03.281506181 +0000 UTC m=+1552.923548170" watchObservedRunningTime="2026-09-09 15:46:03.282072808 +0000 UTC m=+1552.924114843"
Sep 09 15:46:03.576572 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:03.576498    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:03.635764 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:03.635729    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:03.822763 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:03.822732    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:03.949579 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:03.949551    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:08.975443 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:08.975404    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9" podUID="b7522379-2fe7-483d-b5b9-dafe8f37dcc5" containerName="main" containerID="cri-o://78794fb17c081c037b4aef81c1018d9edb5da2febb71603943c8af98da22f9c5" gracePeriod=45
Sep 09 15:46:08.978704 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:08.978681    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:09.221943 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.221919    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:46:09.269699 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.269629    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-model-cache\") pod \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") "
Sep 09 15:46:09.269699 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.269671    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqtb8\" (UniqueName: \"kubernetes.io/projected/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-kube-api-access-vqtb8\") pod \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") "
Sep 09 15:46:09.269699 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.269692    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-dshm\") pod \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") "
Sep 09 15:46:09.269946 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.269710    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-tmp-dir\") pod \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") "
Sep 09 15:46:09.269946 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.269858    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7522379-2fe7-483d-b5b9-dafe8f37dcc5" (UID: "b7522379-2fe7-483d-b5b9-dafe8f37dcc5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:46:09.269946 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.269888    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-tls-certs\") pod \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") "
Sep 09 15:46:09.269946 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.269929    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7522379-2fe7-483d-b5b9-dafe8f37dcc5" (UID: "b7522379-2fe7-483d-b5b9-dafe8f37dcc5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:46:09.269946 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.269937    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-home\") pod \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\" (UID: \"b7522379-2fe7-483d-b5b9-dafe8f37dcc5\") "
Sep 09 15:46:09.270220 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.270139    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-home" (OuterVolumeSpecName: "home") pod "b7522379-2fe7-483d-b5b9-dafe8f37dcc5" (UID: "b7522379-2fe7-483d-b5b9-dafe8f37dcc5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:46:09.270434 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.270396    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-model-cache\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:46:09.270434 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.270421    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-tmp-dir\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:46:09.270434 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.270433    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-home\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:46:09.272092 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.272065    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-dshm" (OuterVolumeSpecName: "dshm") pod "b7522379-2fe7-483d-b5b9-dafe8f37dcc5" (UID: "b7522379-2fe7-483d-b5b9-dafe8f37dcc5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:46:09.272092 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.272074    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-kube-api-access-vqtb8" (OuterVolumeSpecName: "kube-api-access-vqtb8") pod "b7522379-2fe7-483d-b5b9-dafe8f37dcc5" (UID: "b7522379-2fe7-483d-b5b9-dafe8f37dcc5"). InnerVolumeSpecName "kube-api-access-vqtb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:46:09.272234 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.272105    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7522379-2fe7-483d-b5b9-dafe8f37dcc5" (UID: "b7522379-2fe7-483d-b5b9-dafe8f37dcc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:46:09.301205 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.301181    2578 generic.go:356] "Generic (PLEG): container finished" podID="b7522379-2fe7-483d-b5b9-dafe8f37dcc5" containerID="78794fb17c081c037b4aef81c1018d9edb5da2febb71603943c8af98da22f9c5" exitCode=0
Sep 09 15:46:09.301322 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.301223    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"
Sep 09 15:46:09.301322 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.301235    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9" event={"ID":"b7522379-2fe7-483d-b5b9-dafe8f37dcc5","Type":"ContainerDied","Data":"78794fb17c081c037b4aef81c1018d9edb5da2febb71603943c8af98da22f9c5"}
Sep 09 15:46:09.301322 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.301289    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9" event={"ID":"b7522379-2fe7-483d-b5b9-dafe8f37dcc5","Type":"ContainerDied","Data":"754e8ba3404d729ff20dce059ff2d0a7009b040e9e8780fa63fc62fc04ab6a43"}
Sep 09 15:46:09.301322 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.301313    2578 scope.go:117] "RemoveContainer" containerID="78794fb17c081c037b4aef81c1018d9edb5da2febb71603943c8af98da22f9c5"
Sep 09 15:46:09.312961 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.312945    2578 scope.go:117] "RemoveContainer" containerID="78794fb17c081c037b4aef81c1018d9edb5da2febb71603943c8af98da22f9c5"
Sep 09 15:46:09.313274 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:46:09.313254    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78794fb17c081c037b4aef81c1018d9edb5da2febb71603943c8af98da22f9c5\": container with ID starting with 78794fb17c081c037b4aef81c1018d9edb5da2febb71603943c8af98da22f9c5 not found: ID does not exist" containerID="78794fb17c081c037b4aef81c1018d9edb5da2febb71603943c8af98da22f9c5"
Sep 09 15:46:09.313351 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.313281    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78794fb17c081c037b4aef81c1018d9edb5da2febb71603943c8af98da22f9c5"} err="failed to get container status \"78794fb17c081c037b4aef81c1018d9edb5da2febb71603943c8af98da22f9c5\": rpc error: code = NotFound desc = could not find container \"78794fb17c081c037b4aef81c1018d9edb5da2febb71603943c8af98da22f9c5\": container with ID starting with 78794fb17c081c037b4aef81c1018d9edb5da2febb71603943c8af98da22f9c5 not found: ID does not exist"
Sep 09 15:46:09.324774 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.324753    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"]
Sep 09 15:46:09.328305 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.328285    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-267962/test-model-267962-kserve-c958c5d69-qkpl9"]
Sep 09 15:46:09.371758 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.371735    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqtb8\" (UniqueName: \"kubernetes.io/projected/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-kube-api-access-vqtb8\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:46:09.371758 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.371757    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-dshm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:46:09.371912 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:09.371766    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7522379-2fe7-483d-b5b9-dafe8f37dcc5-tls-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:46:10.868314 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:10.868278    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7522379-2fe7-483d-b5b9-dafe8f37dcc5" path="/var/lib/kubelet/pods/b7522379-2fe7-483d-b5b9-dafe8f37dcc5/volumes"
Sep 09 15:46:12.173679 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:12.173644    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:12.174201 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:12.173696    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:12.186703 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:12.186678    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:12.329752 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:12.329724    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:46:24.278406 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:24.278367    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:27.494122 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:27.494087    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:27.555128 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:27.555098    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:40.420756 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:40.420689    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:40.727430 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:40.727349    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:41.374802 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:41.374770    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:41.708684 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:41.708657    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:42.067134 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.067055    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"]
Sep 09 15:46:42.067503 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.067491    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7522379-2fe7-483d-b5b9-dafe8f37dcc5" containerName="main"
Sep 09 15:46:42.067546 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.067505    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7522379-2fe7-483d-b5b9-dafe8f37dcc5" containerName="main"
Sep 09 15:46:42.067588 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.067579    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7522379-2fe7-483d-b5b9-dafe8f37dcc5" containerName="main"
Sep 09 15:46:42.070666 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.070647    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.073158 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.073138    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-df30be\"/\"unconfigured-w0-df30be-kserve-self-signed-certs\""
Sep 09 15:46:42.082532 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.082512    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"]
Sep 09 15:46:42.096564 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.096542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-home\") pod \"unconfigured-w0-df30be-kserve-55bf795944-r7j4d\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") " pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.096664 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.096596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-dshm\") pod \"unconfigured-w0-df30be-kserve-55bf795944-r7j4d\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") " pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.096706 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.096679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-tls-certs\") pod \"unconfigured-w0-df30be-kserve-55bf795944-r7j4d\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") " pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.096746 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.096724    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-model-cache\") pod \"unconfigured-w0-df30be-kserve-55bf795944-r7j4d\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") " pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.096853 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.096836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm46l\" (UniqueName: \"kubernetes.io/projected/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-kube-api-access-zm46l\") pod \"unconfigured-w0-df30be-kserve-55bf795944-r7j4d\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") " pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.096890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.096877    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-tmp-dir\") pod \"unconfigured-w0-df30be-kserve-55bf795944-r7j4d\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") " pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.197841 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.197805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-dshm\") pod \"unconfigured-w0-df30be-kserve-55bf795944-r7j4d\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") " pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.198045 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.197865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-tls-certs\") pod \"unconfigured-w0-df30be-kserve-55bf795944-r7j4d\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") " pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.198045 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.197888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-model-cache\") pod \"unconfigured-w0-df30be-kserve-55bf795944-r7j4d\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") " pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.198045 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.197951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm46l\" (UniqueName: \"kubernetes.io/projected/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-kube-api-access-zm46l\") pod \"unconfigured-w0-df30be-kserve-55bf795944-r7j4d\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") " pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.198045 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.198011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-tmp-dir\") pod \"unconfigured-w0-df30be-kserve-55bf795944-r7j4d\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") " pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.198275 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.198063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-home\") pod \"unconfigured-w0-df30be-kserve-55bf795944-r7j4d\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") " pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.198383 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.198362    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-model-cache\") pod \"unconfigured-w0-df30be-kserve-55bf795944-r7j4d\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") " pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.198424 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.198401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-tmp-dir\") pod \"unconfigured-w0-df30be-kserve-55bf795944-r7j4d\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") " pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.198471 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.198456    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-home\") pod \"unconfigured-w0-df30be-kserve-55bf795944-r7j4d\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") " pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.200442 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.200413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-dshm\") pod \"unconfigured-w0-df30be-kserve-55bf795944-r7j4d\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") " pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.200599 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.200580    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-tls-certs\") pod \"unconfigured-w0-df30be-kserve-55bf795944-r7j4d\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") " pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.207002 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.206963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm46l\" (UniqueName: \"kubernetes.io/projected/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-kube-api-access-zm46l\") pod \"unconfigured-w0-df30be-kserve-55bf795944-r7j4d\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") " pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.380194 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.380105    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:42.651814 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:42.651786    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"]
Sep 09 15:46:42.653053 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:46:42.653024    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97d1bc49_2bac_4edf_a2a8_b953d81fcb9f.slice/crio-14af3faef2899b09aa994ec6ed2773463658781404d1a4e07ff77f9b7111c0c1 WatchSource:0}: Error finding container 14af3faef2899b09aa994ec6ed2773463658781404d1a4e07ff77f9b7111c0c1: Status 404 returned error can't find the container with id 14af3faef2899b09aa994ec6ed2773463658781404d1a4e07ff77f9b7111c0c1
Sep 09 15:46:43.487677 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:43.487640    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d" event={"ID":"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f","Type":"ContainerStarted","Data":"7f6f29ee50f4146173d80fbf12bb3db8e2356664d5b63a9cbee21e07d89ee3c8"}
Sep 09 15:46:43.488062 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:43.487687    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d" event={"ID":"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f","Type":"ContainerStarted","Data":"14af3faef2899b09aa994ec6ed2773463658781404d1a4e07ff77f9b7111c0c1"}
Sep 09 15:46:43.504961 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:43.504912    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d" podStartSLOduration=1.504896069 podStartE2EDuration="1.504896069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:46:43.503213465 +0000 UTC m=+1593.145255461" watchObservedRunningTime="2026-09-09 15:46:43.504896069 +0000 UTC m=+1593.146938057"
Sep 09 15:46:45.067291 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:45.067246    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:45.165725 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:45.165693    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:46.049693 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.042044    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"]
Sep 09 15:46:46.049693 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.048171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 15:46:46.056013 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.055574    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-450a51-serving-cert\""
Sep 09 15:46:46.057170 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.057147    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"]
Sep 09 15:46:46.073404 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.073372    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:46.139208 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.139180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b36a70e-acc8-4a14-89ff-262b0c9048ad-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p\" (UID: \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 15:46:46.139392 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.139234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8g6b\" (UniqueName: \"kubernetes.io/projected/6b36a70e-acc8-4a14-89ff-262b0c9048ad-kube-api-access-n8g6b\") pod \"maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p\" (UID: \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 15:46:46.139392 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.139301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b36a70e-acc8-4a14-89ff-262b0c9048ad-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p\" (UID: \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 15:46:46.139392 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.139347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b36a70e-acc8-4a14-89ff-262b0c9048ad-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p\" (UID: \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 15:46:46.139929 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.139909    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:46.228443 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.228410    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:46.240040 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.240000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b36a70e-acc8-4a14-89ff-262b0c9048ad-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p\" (UID: \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 15:46:46.240176 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.240077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8g6b\" (UniqueName: \"kubernetes.io/projected/6b36a70e-acc8-4a14-89ff-262b0c9048ad-kube-api-access-n8g6b\") pod \"maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p\" (UID: \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 15:46:46.240176 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.240109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b36a70e-acc8-4a14-89ff-262b0c9048ad-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p\" (UID: \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 15:46:46.240176 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.240151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b36a70e-acc8-4a14-89ff-262b0c9048ad-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p\" (UID: \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 15:46:46.240351 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:46:46.240318    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-serving-cert: secret "maas-api-e2e-shared-a-w3-450a51-serving-cert" not found
Sep 09 15:46:46.240407 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:46:46.240394    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b36a70e-acc8-4a14-89ff-262b0c9048ad-maas-api-tls podName:6b36a70e-acc8-4a14-89ff-262b0c9048ad nodeName:}" failed. No retries permitted until 2026-09-09 15:46:46.740374382 +0000 UTC m=+1596.382416363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6b36a70e-acc8-4a14-89ff-262b0c9048ad-maas-api-tls") pod "maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p" (UID: "6b36a70e-acc8-4a14-89ff-262b0c9048ad") : secret "maas-api-e2e-shared-a-w3-450a51-serving-cert" not found
Sep 09 15:46:46.240892 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.240870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b36a70e-acc8-4a14-89ff-262b0c9048ad-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p\" (UID: \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 15:46:46.242749 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.242721    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b36a70e-acc8-4a14-89ff-262b0c9048ad-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p\" (UID: \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 15:46:46.253656 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.253628    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8g6b\" (UniqueName: \"kubernetes.io/projected/6b36a70e-acc8-4a14-89ff-262b0c9048ad-kube-api-access-n8g6b\") pod \"maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p\" (UID: \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 15:46:46.485183 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.485151    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:46.599916 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.599883    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:46.694131 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.694099    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:46:46.746157 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.746081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b36a70e-acc8-4a14-89ff-262b0c9048ad-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p\" (UID: \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 15:46:46.748614 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.748594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b36a70e-acc8-4a14-89ff-262b0c9048ad-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p\" (UID: \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 15:46:46.965589 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:46.965559    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 15:46:47.240449 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:47.240419    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"]
Sep 09 15:46:47.242234 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:46:47.242195    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b36a70e_acc8_4a14_89ff_262b0c9048ad.slice/crio-3ba80407b8c8482481d87985260d26201ee561d0b42e097939369320c25a1ceb WatchSource:0}: Error finding container 3ba80407b8c8482481d87985260d26201ee561d0b42e097939369320c25a1ceb: Status 404 returned error can't find the container with id 3ba80407b8c8482481d87985260d26201ee561d0b42e097939369320c25a1ceb
Sep 09 15:46:47.511127 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:47.511088    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p" event={"ID":"6b36a70e-acc8-4a14-89ff-262b0c9048ad","Type":"ContainerStarted","Data":"3ba80407b8c8482481d87985260d26201ee561d0b42e097939369320c25a1ceb"}
Sep 09 15:46:48.518268 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:48.518235    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p" event={"ID":"6b36a70e-acc8-4a14-89ff-262b0c9048ad","Type":"ContainerStarted","Data":"e6371b6da2f24c22a255e2c54a5c32acf35e3869f87705c38ebfb547a99dfc4d"}
Sep 09 15:46:48.518720 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:48.518299    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 15:46:48.537164 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:48.537110    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p" podStartSLOduration=2.1304905339999998 podStartE2EDuration="2.537094926s" totalImagesPullingTime="406.604392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:46:48.535480609 +0000 UTC m=+1598.177522600" watchObservedRunningTime="2026-09-09 15:46:48.537094926 +0000 UTC m=+1598.179136969"
Sep 09 15:46:52.381206 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:52.381176    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:52.381600 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:52.381219    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:52.393607 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:52.393579    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:52.551680 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:52.551654    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:46:54.529875 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:46:54.529849    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 15:47:01.970270 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:01.970192    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:10.140395 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:10.140340    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:10.533404 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:10.533290    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:10.843224 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:10.843115    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:11.019458 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:11.019409    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:14.341154 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:14.341118    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:14.473153 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:14.473127    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:15.562793 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:15.562758    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:15.625863 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:15.625828    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:15.778718 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:15.778685    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:16.122803 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:16.122777    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:16.222301 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:16.222262    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:24.775167 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:24.775127    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:36.574059 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:36.574018    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:37.606965 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:37.606928    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"]
Sep 09 15:47:37.607393 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:37.607210    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k" podUID="eaa99199-675b-4029-8f81-a0896035d284" containerName="maas-api" containerID="cri-o://8253f80ccf45b8a1717dcf8f57a25f0e65a76d31a21ebee41eaaa64c92a562ba" gracePeriod=30
Sep 09 15:47:37.631926 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:37.631890    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:37.727361 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:37.727323    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:37.788968 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:37.788936    2578 generic.go:356] "Generic (PLEG): container finished" podID="eaa99199-675b-4029-8f81-a0896035d284" containerID="8253f80ccf45b8a1717dcf8f57a25f0e65a76d31a21ebee41eaaa64c92a562ba" exitCode=0
Sep 09 15:47:37.789166 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:37.789141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k" event={"ID":"eaa99199-675b-4029-8f81-a0896035d284","Type":"ContainerDied","Data":"8253f80ccf45b8a1717dcf8f57a25f0e65a76d31a21ebee41eaaa64c92a562ba"}
Sep 09 15:47:37.826615 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:37.826576    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:37.874455 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:37.874395    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"
Sep 09 15:47:37.899700 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:37.899669    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:37.971158 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:37.971120    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaa99199-675b-4029-8f81-a0896035d284-metrics-certs\") pod \"eaa99199-675b-4029-8f81-a0896035d284\" (UID: \"eaa99199-675b-4029-8f81-a0896035d284\") "
Sep 09 15:47:37.971365 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:37.971268    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa99199-675b-4029-8f81-a0896035d284-maas-api-tls\") pod \"eaa99199-675b-4029-8f81-a0896035d284\" (UID: \"eaa99199-675b-4029-8f81-a0896035d284\") "
Sep 09 15:47:37.971365 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:37.971291    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaa99199-675b-4029-8f81-a0896035d284-openshift-service-ca\") pod \"eaa99199-675b-4029-8f81-a0896035d284\" (UID: \"eaa99199-675b-4029-8f81-a0896035d284\") "
Sep 09 15:47:37.971365 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:37.971312    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4cff\" (UniqueName: \"kubernetes.io/projected/eaa99199-675b-4029-8f81-a0896035d284-kube-api-access-m4cff\") pod \"eaa99199-675b-4029-8f81-a0896035d284\" (UID: \"eaa99199-675b-4029-8f81-a0896035d284\") "
Sep 09 15:47:37.971733 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:37.971690    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa99199-675b-4029-8f81-a0896035d284-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eaa99199-675b-4029-8f81-a0896035d284" (UID: "eaa99199-675b-4029-8f81-a0896035d284"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:47:37.972283 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:37.972118    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaa99199-675b-4029-8f81-a0896035d284-openshift-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:47:37.973822 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:37.973793    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaa99199-675b-4029-8f81-a0896035d284-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eaa99199-675b-4029-8f81-a0896035d284" (UID: "eaa99199-675b-4029-8f81-a0896035d284"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:47:37.974177 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:37.974144    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaa99199-675b-4029-8f81-a0896035d284-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eaa99199-675b-4029-8f81-a0896035d284" (UID: "eaa99199-675b-4029-8f81-a0896035d284"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:47:37.974337 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:37.974302    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa99199-675b-4029-8f81-a0896035d284-kube-api-access-m4cff" (OuterVolumeSpecName: "kube-api-access-m4cff") pod "eaa99199-675b-4029-8f81-a0896035d284" (UID: "eaa99199-675b-4029-8f81-a0896035d284"). InnerVolumeSpecName "kube-api-access-m4cff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:47:38.073525 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:38.073485    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaa99199-675b-4029-8f81-a0896035d284-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:47:38.073525 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:38.073513    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa99199-675b-4029-8f81-a0896035d284-maas-api-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:47:38.073525 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:38.073522    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4cff\" (UniqueName: \"kubernetes.io/projected/eaa99199-675b-4029-8f81-a0896035d284-kube-api-access-m4cff\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:47:38.797492 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:38.797465    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"
Sep 09 15:47:38.797942 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:38.797504    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k" event={"ID":"eaa99199-675b-4029-8f81-a0896035d284","Type":"ContainerDied","Data":"d781cad1365f39dbb70ea73991755deb38c5c4a165d5319659b9740062914e7d"}
Sep 09 15:47:38.797942 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:38.797580    2578 scope.go:117] "RemoveContainer" containerID="8253f80ccf45b8a1717dcf8f57a25f0e65a76d31a21ebee41eaaa64c92a562ba"
Sep 09 15:47:38.822871 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:38.822841    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"]
Sep 09 15:47:38.827817 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:38.827795    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aeb0f8ca-694df77bb4-z9s8k"]
Sep 09 15:47:38.869257 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:38.869225    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaa99199-675b-4029-8f81-a0896035d284" path="/var/lib/kubelet/pods/eaa99199-675b-4029-8f81-a0896035d284/volumes"
Sep 09 15:47:39.417743 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:39.417713    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:41.786936 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:41.786901    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:47:46.776526 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:46.776494    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"]
Sep 09 15:47:46.871766 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:47:46.871733    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:48:00.948720 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:00.948685    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:48:01.137917 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:01.137864    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:48:01.796902 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:01.796868    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g" podUID="92791d59-620c-4aef-a8eb-65cd4db02b34" containerName="main" containerID="cri-o://8e828d3cde7407403af5c56290b8def970d43ff9533935dc245998ee7d9b3443" gracePeriod=45
Sep 09 15:48:01.922735 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:01.922705    2578 generic.go:356] "Generic (PLEG): container finished" podID="92791d59-620c-4aef-a8eb-65cd4db02b34" containerID="8e828d3cde7407403af5c56290b8def970d43ff9533935dc245998ee7d9b3443" exitCode=0
Sep 09 15:48:01.922900 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:01.922783    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g" event={"ID":"92791d59-620c-4aef-a8eb-65cd4db02b34","Type":"ContainerDied","Data":"8e828d3cde7407403af5c56290b8def970d43ff9533935dc245998ee7d9b3443"}
Sep 09 15:48:02.059768 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.059709    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:48:02.119456 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.119429    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-dshm\") pod \"92791d59-620c-4aef-a8eb-65cd4db02b34\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") "
Sep 09 15:48:02.119627 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.119496    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92791d59-620c-4aef-a8eb-65cd4db02b34-tls-certs\") pod \"92791d59-620c-4aef-a8eb-65cd4db02b34\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") "
Sep 09 15:48:02.119680 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.119652    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-home\") pod \"92791d59-620c-4aef-a8eb-65cd4db02b34\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") "
Sep 09 15:48:02.119769 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.119754    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvpw2\" (UniqueName: \"kubernetes.io/projected/92791d59-620c-4aef-a8eb-65cd4db02b34-kube-api-access-mvpw2\") pod \"92791d59-620c-4aef-a8eb-65cd4db02b34\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") "
Sep 09 15:48:02.119834 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.119814    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-tmp-dir\") pod \"92791d59-620c-4aef-a8eb-65cd4db02b34\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") "
Sep 09 15:48:02.119890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.119850    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-model-cache\") pod \"92791d59-620c-4aef-a8eb-65cd4db02b34\" (UID: \"92791d59-620c-4aef-a8eb-65cd4db02b34\") "
Sep 09 15:48:02.120008 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.119942    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-home" (OuterVolumeSpecName: "home") pod "92791d59-620c-4aef-a8eb-65cd4db02b34" (UID: "92791d59-620c-4aef-a8eb-65cd4db02b34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:48:02.120377 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.120173    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "92791d59-620c-4aef-a8eb-65cd4db02b34" (UID: "92791d59-620c-4aef-a8eb-65cd4db02b34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:48:02.120377 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.120299    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-tmp-dir\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:48:02.120547 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.120382    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-home\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:48:02.120547 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.120443    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-model-cache" (OuterVolumeSpecName: "model-cache") pod "92791d59-620c-4aef-a8eb-65cd4db02b34" (UID: "92791d59-620c-4aef-a8eb-65cd4db02b34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:48:02.122462 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.122440    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-dshm" (OuterVolumeSpecName: "dshm") pod "92791d59-620c-4aef-a8eb-65cd4db02b34" (UID: "92791d59-620c-4aef-a8eb-65cd4db02b34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:48:02.122794 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.122754    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92791d59-620c-4aef-a8eb-65cd4db02b34-kube-api-access-mvpw2" (OuterVolumeSpecName: "kube-api-access-mvpw2") pod "92791d59-620c-4aef-a8eb-65cd4db02b34" (UID: "92791d59-620c-4aef-a8eb-65cd4db02b34"). InnerVolumeSpecName "kube-api-access-mvpw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:48:02.123094 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.123068    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92791d59-620c-4aef-a8eb-65cd4db02b34-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92791d59-620c-4aef-a8eb-65cd4db02b34" (UID: "92791d59-620c-4aef-a8eb-65cd4db02b34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:48:02.221293 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.221259    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-dshm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:48:02.221293 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.221287    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92791d59-620c-4aef-a8eb-65cd4db02b34-tls-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:48:02.221293 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.221297    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvpw2\" (UniqueName: \"kubernetes.io/projected/92791d59-620c-4aef-a8eb-65cd4db02b34-kube-api-access-mvpw2\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:48:02.221540 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.221307    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92791d59-620c-4aef-a8eb-65cd4db02b34-model-cache\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:48:02.929780 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.929748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g" event={"ID":"92791d59-620c-4aef-a8eb-65cd4db02b34","Type":"ContainerDied","Data":"27d3d9c0ef02bec288efd225945a54170d464fc5f85c0e901c48d3509de51b55"}
Sep 09 15:48:02.929780 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.929778    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"
Sep 09 15:48:02.930034 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.929799    2578 scope.go:117] "RemoveContainer" containerID="8e828d3cde7407403af5c56290b8def970d43ff9533935dc245998ee7d9b3443"
Sep 09 15:48:02.949039 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.949013    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"]
Sep 09 15:48:02.953341 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:02.953315    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-aeb0f8ca/e2e-lifecycle-model-aeb0f8ca-kserve-5745d9f667-wfw5g"]
Sep 09 15:48:03.564020 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:03.563960    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:48:04.868399 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:04.868369    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92791d59-620c-4aef-a8eb-65cd4db02b34" path="/var/lib/kubelet/pods/92791d59-620c-4aef-a8eb-65cd4db02b34/volumes"
Sep 09 15:48:14.464911 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.464878    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"]
Sep 09 15:48:14.465378 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.465368    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92791d59-620c-4aef-a8eb-65cd4db02b34" containerName="main"
Sep 09 15:48:14.465425 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.465382    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="92791d59-620c-4aef-a8eb-65cd4db02b34" containerName="main"
Sep 09 15:48:14.465425 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.465404    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaa99199-675b-4029-8f81-a0896035d284" containerName="maas-api"
Sep 09 15:48:14.465425 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.465409    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaa99199-675b-4029-8f81-a0896035d284" containerName="maas-api"
Sep 09 15:48:14.465534 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.465491    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="92791d59-620c-4aef-a8eb-65cd4db02b34" containerName="main"
Sep 09 15:48:14.465534 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.465505    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaa99199-675b-4029-8f81-a0896035d284" containerName="maas-api"
Sep 09 15:48:14.468806 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.468788    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.471255 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.471233    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-df30be\"/\"embedding-w0-df30be-kserve-self-signed-certs\""
Sep 09 15:48:14.478844 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.478818    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"]
Sep 09 15:48:14.492802 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.492769    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:48:14.551144 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.551115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-home\") pod \"embedding-w0-df30be-kserve-66b6757b4d-wbh5g\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") " pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.551332 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.551173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-model-cache\") pod \"embedding-w0-df30be-kserve-66b6757b4d-wbh5g\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") " pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.551332 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.551269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhgwk\" (UniqueName: \"kubernetes.io/projected/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-kube-api-access-lhgwk\") pod \"embedding-w0-df30be-kserve-66b6757b4d-wbh5g\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") " pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.551457 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.551333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-tmp-dir\") pod \"embedding-w0-df30be-kserve-66b6757b4d-wbh5g\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") " pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.551457 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.551399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-tls-certs\") pod \"embedding-w0-df30be-kserve-66b6757b4d-wbh5g\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") " pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.551457 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.551443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-dshm\") pod \"embedding-w0-df30be-kserve-66b6757b4d-wbh5g\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") " pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.652050 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.652021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-home\") pod \"embedding-w0-df30be-kserve-66b6757b4d-wbh5g\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") " pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.652217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.652079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-model-cache\") pod \"embedding-w0-df30be-kserve-66b6757b4d-wbh5g\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") " pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.652217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.652111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhgwk\" (UniqueName: \"kubernetes.io/projected/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-kube-api-access-lhgwk\") pod \"embedding-w0-df30be-kserve-66b6757b4d-wbh5g\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") " pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.652217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.652146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-tmp-dir\") pod \"embedding-w0-df30be-kserve-66b6757b4d-wbh5g\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") " pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.652411 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.652286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-tls-certs\") pod \"embedding-w0-df30be-kserve-66b6757b4d-wbh5g\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") " pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.652411 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.652353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-dshm\") pod \"embedding-w0-df30be-kserve-66b6757b4d-wbh5g\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") " pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.652520 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.652428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-home\") pod \"embedding-w0-df30be-kserve-66b6757b4d-wbh5g\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") " pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.652520 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.652479    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-model-cache\") pod \"embedding-w0-df30be-kserve-66b6757b4d-wbh5g\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") " pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.652596 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.652531    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-tmp-dir\") pod \"embedding-w0-df30be-kserve-66b6757b4d-wbh5g\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") " pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.654689 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.654667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-dshm\") pod \"embedding-w0-df30be-kserve-66b6757b4d-wbh5g\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") " pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.654960 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.654942    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-tls-certs\") pod \"embedding-w0-df30be-kserve-66b6757b4d-wbh5g\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") " pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.660592 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.660567    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhgwk\" (UniqueName: \"kubernetes.io/projected/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-kube-api-access-lhgwk\") pod \"embedding-w0-df30be-kserve-66b6757b4d-wbh5g\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") " pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:14.778918 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:14.778819    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:15.055002 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:15.054959    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"]
Sep 09 15:48:15.057055 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:48:15.057026    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddae030cb_2e14_41b9_b211_ec0af9f6e9b1.slice/crio-7221ec2b063dfd95e14135f54e858387215cceda4a230358d2afeafa3f31b6a0 WatchSource:0}: Error finding container 7221ec2b063dfd95e14135f54e858387215cceda4a230358d2afeafa3f31b6a0: Status 404 returned error can't find the container with id 7221ec2b063dfd95e14135f54e858387215cceda4a230358d2afeafa3f31b6a0
Sep 09 15:48:16.001442 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:16.001399    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g" event={"ID":"dae030cb-2e14-41b9-b211-ec0af9f6e9b1","Type":"ContainerStarted","Data":"f7b0e658aa884ddf425515e8019d203b02b9c5a70d5a994989d3ecfc3d6bdf79"}
Sep 09 15:48:16.001827 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:16.001455    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g" event={"ID":"dae030cb-2e14-41b9-b211-ec0af9f6e9b1","Type":"ContainerStarted","Data":"7221ec2b063dfd95e14135f54e858387215cceda4a230358d2afeafa3f31b6a0"}
Sep 09 15:48:16.018501 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:16.018392    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g" podStartSLOduration=2.018373466 podStartE2EDuration="2.018373466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:48:16.018167568 +0000 UTC m=+1685.660209557" watchObservedRunningTime="2026-09-09 15:48:16.018373466 +0000 UTC m=+1685.660415456"
Sep 09 15:48:16.101084 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:16.101033    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:48:17.068781 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:17.068751    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:48:23.751354 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:23.751324    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:48:24.404692 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:24.404661    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:48:24.779145 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:24.779035    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:24.779145 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:24.779097    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:24.792487 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:24.792460    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:25.064464 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:25.064383    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:48:29.991199 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:29.991160    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:48:44.371861 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:44.371824    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:48:46.691791 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:48:46.691756    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:12.663096 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:12.663065    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:13.070021 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:13.069886    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:13.591927 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:13.591891    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:13.857338 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:13.857251    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:14.580209 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:14.580172    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:14.742298 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:14.742268    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:16.331920 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:16.331888    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:20.668930 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.668898    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"]
Sep 09 15:49:20.672242 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.672223    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.675437 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.675419    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1e15c8\"/\"embedding-w4-1e15c8-kserve-self-signed-certs\""
Sep 09 15:49:20.684362 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.684342    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"]
Sep 09 15:49:20.717216 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.717184    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:20.841015 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.840946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f94d58-8121-49d0-96ed-4890dd50b00b-tls-certs\") pod \"embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") " pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.841202 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.841094    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-home\") pod \"embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") " pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.841202 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.841152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-model-cache\") pod \"embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") " pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.841202 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.841176    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt2hb\" (UniqueName: \"kubernetes.io/projected/f5f94d58-8121-49d0-96ed-4890dd50b00b-kube-api-access-xt2hb\") pod \"embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") " pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.841345 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.841207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-dshm\") pod \"embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") " pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.841345 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.841242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-tmp-dir\") pod \"embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") " pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.942933 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.942827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f94d58-8121-49d0-96ed-4890dd50b00b-tls-certs\") pod \"embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") " pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.942933 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.942916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-home\") pod \"embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") " pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.943198 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.942952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-model-cache\") pod \"embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") " pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.943198 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.942971    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt2hb\" (UniqueName: \"kubernetes.io/projected/f5f94d58-8121-49d0-96ed-4890dd50b00b-kube-api-access-xt2hb\") pod \"embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") " pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.943198 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.943014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-dshm\") pod \"embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") " pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.943198 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.943053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-tmp-dir\") pod \"embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") " pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.943490 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.943467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-tmp-dir\") pod \"embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") " pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.943560 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.943502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-home\") pod \"embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") " pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.943560 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.943543    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-model-cache\") pod \"embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") " pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.945455 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.945434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-dshm\") pod \"embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") " pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.945609 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.945591    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f94d58-8121-49d0-96ed-4890dd50b00b-tls-certs\") pod \"embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") " pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.953181 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.953150    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt2hb\" (UniqueName: \"kubernetes.io/projected/f5f94d58-8121-49d0-96ed-4890dd50b00b-kube-api-access-xt2hb\") pod \"embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") " pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:20.981683 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:20.981654    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:21.259178 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:21.259153    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"]
Sep 09 15:49:21.261127 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:49:21.261095    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5f94d58_8121_49d0_96ed_4890dd50b00b.slice/crio-2af50c2e66a3523ce476a9a0f36f5a5dd108053fefa8268c347159ba64ea29cc WatchSource:0}: Error finding container 2af50c2e66a3523ce476a9a0f36f5a5dd108053fefa8268c347159ba64ea29cc: Status 404 returned error can't find the container with id 2af50c2e66a3523ce476a9a0f36f5a5dd108053fefa8268c347159ba64ea29cc
Sep 09 15:49:21.364022 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:21.363965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd" event={"ID":"f5f94d58-8121-49d0-96ed-4890dd50b00b","Type":"ContainerStarted","Data":"82766adac26ecd276fd30cbb3a8af0f831b08ed48f6a13a5ee744ed0791d6e59"}
Sep 09 15:49:21.364198 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:21.364034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd" event={"ID":"f5f94d58-8121-49d0-96ed-4890dd50b00b","Type":"ContainerStarted","Data":"2af50c2e66a3523ce476a9a0f36f5a5dd108053fefa8268c347159ba64ea29cc"}
Sep 09 15:49:21.382830 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:21.382783    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd" podStartSLOduration=1.382768929 podStartE2EDuration="1.382768929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:49:21.381739293 +0000 UTC m=+1751.023781280" watchObservedRunningTime="2026-09-09 15:49:21.382768929 +0000 UTC m=+1751.024810919"
Sep 09 15:49:22.758166 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:22.758129    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:22.857513 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:22.857479    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:23.502858 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:23.502828    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:29.912064 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:29.912019    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:29.991939 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:29.991907    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:30.113233 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:30.113202    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:30.508417 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:30.508382    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:30.635178 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:30.635145    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:30.982831 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:30.982792    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:30.983338 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:30.982884    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:30.995625 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:30.995599    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:31.430434 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:31.430396    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:49:40.729878 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:40.729841    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:40.975684 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:40.975651    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:42.901877 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:42.901837    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:54.215575 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:54.215509    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:54.681608 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:54.681577    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:49:59.193548 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:59.193515    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"]
Sep 09 15:49:59.200425 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:59.200396    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"]
Sep 09 15:49:59.209721 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:59.209694    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"]
Sep 09 15:49:59.220351 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:59.220324    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"]
Sep 09 15:49:59.230070 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:59.230044    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"]
Sep 09 15:49:59.235395 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:59.235364    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"]
Sep 09 15:49:59.280661 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:49:59.280633    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:01.956439 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:01.956364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:05.078754 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:05.078714    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:07.974373 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:07.974339    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:08.046680 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:08.046645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:08.381620 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:08.381541    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:08.720769 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:08.720740    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:08.826671 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:08.826645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:10.752999 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:10.752387    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"]
Sep 09 15:50:10.752999 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:10.752625    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw" podUID="7eb684ff-51f2-49c9-89e6-69fc847306cc" containerName="maas-api" containerID="cri-o://a41ef767da8e8a944cb92398c5090fcced5516304a937b990aeeb1d4cf874f96" gracePeriod=30
Sep 09 15:50:11.021057 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.021002    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:50:11.064569 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.064540    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb684ff-51f2-49c9-89e6-69fc847306cc-metrics-certs\") pod \"7eb684ff-51f2-49c9-89e6-69fc847306cc\" (UID: \"7eb684ff-51f2-49c9-89e6-69fc847306cc\") "
Sep 09 15:50:11.064758 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.064637    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eb684ff-51f2-49c9-89e6-69fc847306cc-maas-api-tls\") pod \"7eb684ff-51f2-49c9-89e6-69fc847306cc\" (UID: \"7eb684ff-51f2-49c9-89e6-69fc847306cc\") "
Sep 09 15:50:11.064758 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.064673    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6gz9\" (UniqueName: \"kubernetes.io/projected/7eb684ff-51f2-49c9-89e6-69fc847306cc-kube-api-access-k6gz9\") pod \"7eb684ff-51f2-49c9-89e6-69fc847306cc\" (UID: \"7eb684ff-51f2-49c9-89e6-69fc847306cc\") "
Sep 09 15:50:11.064758 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.064696    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb684ff-51f2-49c9-89e6-69fc847306cc-openshift-service-ca\") pod \"7eb684ff-51f2-49c9-89e6-69fc847306cc\" (UID: \"7eb684ff-51f2-49c9-89e6-69fc847306cc\") "
Sep 09 15:50:11.065131 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.065106    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb684ff-51f2-49c9-89e6-69fc847306cc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7eb684ff-51f2-49c9-89e6-69fc847306cc" (UID: "7eb684ff-51f2-49c9-89e6-69fc847306cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:50:11.066937 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.066912    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eb684ff-51f2-49c9-89e6-69fc847306cc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7eb684ff-51f2-49c9-89e6-69fc847306cc" (UID: "7eb684ff-51f2-49c9-89e6-69fc847306cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:50:11.067216 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.067196    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eb684ff-51f2-49c9-89e6-69fc847306cc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7eb684ff-51f2-49c9-89e6-69fc847306cc" (UID: "7eb684ff-51f2-49c9-89e6-69fc847306cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:50:11.067299 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.067274    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb684ff-51f2-49c9-89e6-69fc847306cc-kube-api-access-k6gz9" (OuterVolumeSpecName: "kube-api-access-k6gz9") pod "7eb684ff-51f2-49c9-89e6-69fc847306cc" (UID: "7eb684ff-51f2-49c9-89e6-69fc847306cc"). InnerVolumeSpecName "kube-api-access-k6gz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:50:11.166533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.166503    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb684ff-51f2-49c9-89e6-69fc847306cc-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:11.166533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.166529    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eb684ff-51f2-49c9-89e6-69fc847306cc-maas-api-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:11.166533 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.166538    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6gz9\" (UniqueName: \"kubernetes.io/projected/7eb684ff-51f2-49c9-89e6-69fc847306cc-kube-api-access-k6gz9\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:11.166815 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.166549    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb684ff-51f2-49c9-89e6-69fc847306cc-openshift-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:11.170015 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.169966    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:11.576773 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.576739    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:11.637881 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.637850    2578 generic.go:356] "Generic (PLEG): container finished" podID="7eb684ff-51f2-49c9-89e6-69fc847306cc" containerID="a41ef767da8e8a944cb92398c5090fcced5516304a937b990aeeb1d4cf874f96" exitCode=0
Sep 09 15:50:11.638042 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.637894    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"
Sep 09 15:50:11.638042 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.637910    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw" event={"ID":"7eb684ff-51f2-49c9-89e6-69fc847306cc","Type":"ContainerDied","Data":"a41ef767da8e8a944cb92398c5090fcced5516304a937b990aeeb1d4cf874f96"}
Sep 09 15:50:11.638042 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.637950    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw" event={"ID":"7eb684ff-51f2-49c9-89e6-69fc847306cc","Type":"ContainerDied","Data":"e4b0d3e2beb836de8d4985e7698ea47072ee1f99d1d42ff182c5dc71ec838032"}
Sep 09 15:50:11.638042 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.637972    2578 scope.go:117] "RemoveContainer" containerID="a41ef767da8e8a944cb92398c5090fcced5516304a937b990aeeb1d4cf874f96"
Sep 09 15:50:11.642754 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.642733    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:11.650593 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.650577    2578 scope.go:117] "RemoveContainer" containerID="a41ef767da8e8a944cb92398c5090fcced5516304a937b990aeeb1d4cf874f96"
Sep 09 15:50:11.650838 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:50:11.650818    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41ef767da8e8a944cb92398c5090fcced5516304a937b990aeeb1d4cf874f96\": container with ID starting with a41ef767da8e8a944cb92398c5090fcced5516304a937b990aeeb1d4cf874f96 not found: ID does not exist" containerID="a41ef767da8e8a944cb92398c5090fcced5516304a937b990aeeb1d4cf874f96"
Sep 09 15:50:11.650957 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.650844    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41ef767da8e8a944cb92398c5090fcced5516304a937b990aeeb1d4cf874f96"} err="failed to get container status \"a41ef767da8e8a944cb92398c5090fcced5516304a937b990aeeb1d4cf874f96\": rpc error: code = NotFound desc = could not find container \"a41ef767da8e8a944cb92398c5090fcced5516304a937b990aeeb1d4cf874f96\": container with ID starting with a41ef767da8e8a944cb92398c5090fcced5516304a937b990aeeb1d4cf874f96 not found: ID does not exist"
Sep 09 15:50:11.659461 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.659431    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"]
Sep 09 15:50:11.661506 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:11.661487    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-df30be-c4ccdc8f-8csrw"]
Sep 09 15:50:12.657250 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.657217    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"]
Sep 09 15:50:12.657659 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.657647    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eb684ff-51f2-49c9-89e6-69fc847306cc" containerName="maas-api"
Sep 09 15:50:12.657659 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.657660    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eb684ff-51f2-49c9-89e6-69fc847306cc" containerName="maas-api"
Sep 09 15:50:12.657746 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.657737    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eb684ff-51f2-49c9-89e6-69fc847306cc" containerName="maas-api"
Sep 09 15:50:12.660680 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.660664    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"
Sep 09 15:50:12.664237 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.664219    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3ac2fa-serving-cert\""
Sep 09 15:50:12.673450 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.673428    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"]
Sep 09 15:50:12.711635 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.711612    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:12.781918 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.781889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9688bbf2-a894-4955-929e-3d6d6f1726b5-metrics-certs\") pod \"maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6\" (UID: \"9688bbf2-a894-4955-929e-3d6d6f1726b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"
Sep 09 15:50:12.782110 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.781947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwmgd\" (UniqueName: \"kubernetes.io/projected/9688bbf2-a894-4955-929e-3d6d6f1726b5-kube-api-access-zwmgd\") pod \"maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6\" (UID: \"9688bbf2-a894-4955-929e-3d6d6f1726b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"
Sep 09 15:50:12.782110 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.782010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9688bbf2-a894-4955-929e-3d6d6f1726b5-maas-api-tls\") pod \"maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6\" (UID: \"9688bbf2-a894-4955-929e-3d6d6f1726b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"
Sep 09 15:50:12.782110 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.782074    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9688bbf2-a894-4955-929e-3d6d6f1726b5-openshift-service-ca\") pod \"maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6\" (UID: \"9688bbf2-a894-4955-929e-3d6d6f1726b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"
Sep 09 15:50:12.811242 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.811216    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:12.876792 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.876751    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eb684ff-51f2-49c9-89e6-69fc847306cc" path="/var/lib/kubelet/pods/7eb684ff-51f2-49c9-89e6-69fc847306cc/volumes"
Sep 09 15:50:12.882686 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.882663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9688bbf2-a894-4955-929e-3d6d6f1726b5-openshift-service-ca\") pod \"maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6\" (UID: \"9688bbf2-a894-4955-929e-3d6d6f1726b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"
Sep 09 15:50:12.882804 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.882787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9688bbf2-a894-4955-929e-3d6d6f1726b5-metrics-certs\") pod \"maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6\" (UID: \"9688bbf2-a894-4955-929e-3d6d6f1726b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"
Sep 09 15:50:12.882856 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.882843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwmgd\" (UniqueName: \"kubernetes.io/projected/9688bbf2-a894-4955-929e-3d6d6f1726b5-kube-api-access-zwmgd\") pod \"maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6\" (UID: \"9688bbf2-a894-4955-929e-3d6d6f1726b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"
Sep 09 15:50:12.882927 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.882884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9688bbf2-a894-4955-929e-3d6d6f1726b5-maas-api-tls\") pod \"maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6\" (UID: \"9688bbf2-a894-4955-929e-3d6d6f1726b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"
Sep 09 15:50:12.883506 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.883441    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9688bbf2-a894-4955-929e-3d6d6f1726b5-openshift-service-ca\") pod \"maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6\" (UID: \"9688bbf2-a894-4955-929e-3d6d6f1726b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"
Sep 09 15:50:12.885376 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.885357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9688bbf2-a894-4955-929e-3d6d6f1726b5-maas-api-tls\") pod \"maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6\" (UID: \"9688bbf2-a894-4955-929e-3d6d6f1726b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"
Sep 09 15:50:12.885488 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.885466    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9688bbf2-a894-4955-929e-3d6d6f1726b5-metrics-certs\") pod \"maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6\" (UID: \"9688bbf2-a894-4955-929e-3d6d6f1726b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"
Sep 09 15:50:12.892153 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.892132    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwmgd\" (UniqueName: \"kubernetes.io/projected/9688bbf2-a894-4955-929e-3d6d6f1726b5-kube-api-access-zwmgd\") pod \"maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6\" (UID: \"9688bbf2-a894-4955-929e-3d6d6f1726b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"
Sep 09 15:50:12.911990 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.911918    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:12.974232 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:12.974199    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"
Sep 09 15:50:13.195305 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:13.195223    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:13.250783 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:13.250759    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"]
Sep 09 15:50:13.253230 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:50:13.253203    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9688bbf2_a894_4955_929e_3d6d6f1726b5.slice/crio-ca30022b3ed8fcbbc2d0e76c950181024d58ea9149f9f06dc38684908988cb6a WatchSource:0}: Error finding container ca30022b3ed8fcbbc2d0e76c950181024d58ea9149f9f06dc38684908988cb6a: Status 404 returned error can't find the container with id ca30022b3ed8fcbbc2d0e76c950181024d58ea9149f9f06dc38684908988cb6a
Sep 09 15:50:13.254769 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:13.254753    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 15:50:13.292463 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:13.292435    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:13.651040 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:13.651007    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6" event={"ID":"9688bbf2-a894-4955-929e-3d6d6f1726b5","Type":"ContainerStarted","Data":"ca30022b3ed8fcbbc2d0e76c950181024d58ea9149f9f06dc38684908988cb6a"}
Sep 09 15:50:14.216729 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.216673    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg" podUID="6f69e13f-5e3a-4cac-a449-72458cb37ccc" containerName="main" containerID="cri-o://aa3fcc36cd98b1e1c6d36463eb4ed0214c1539864b4823eb6896c151db028d8c" gracePeriod=45
Sep 09 15:50:14.221420 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.221392    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2" podUID="dccfee92-8984-45fd-899b-7ab786c9525b" containerName="main" containerID="cri-o://586338c5d152b213cb89739e85a0237d6f08018f4ffd383f69116d2271242b57" gracePeriod=45
Sep 09 15:50:14.230159 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.230128    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g" podUID="dae030cb-2e14-41b9-b211-ec0af9f6e9b1" containerName="main" containerID="cri-o://f7b0e658aa884ddf425515e8019d203b02b9c5a70d5a994989d3ecfc3d6bdf79" gracePeriod=45
Sep 09 15:50:14.241036 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.241005    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv" podUID="861de506-45f1-4566-a610-ecf1f980202f" containerName="main" containerID="cri-o://97be43e9ea830385e684e81a0ad0b8e6614c0906a377547f5a041777d5184c7e" gracePeriod=45
Sep 09 15:50:14.253952 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.253924    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h" podUID="fbf28447-9239-4a6e-adfe-7baa6e38ece4" containerName="main" containerID="cri-o://2a8a7bc158d79110cfe838879f1a542c9f38e30dd935fbd22238a1d741028cdd" gracePeriod=45
Sep 09 15:50:14.265325 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.265297    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d" podUID="97d1bc49-2bac-4edf-a2a8-b953d81fcb9f" containerName="main" containerID="cri-o://7f6f29ee50f4146173d80fbf12bb3db8e2356664d5b63a9cbee21e07d89ee3c8" gracePeriod=45
Sep 09 15:50:14.591759 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.584661    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:50:14.591759 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.588943    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:50:14.624537 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.624495    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:50:14.629246 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.629226    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:50:14.633179 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.633157    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:50:14.636719 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.636699    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:50:14.659745 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.659710    2578 generic.go:356] "Generic (PLEG): container finished" podID="861de506-45f1-4566-a610-ecf1f980202f" containerID="97be43e9ea830385e684e81a0ad0b8e6614c0906a377547f5a041777d5184c7e" exitCode=0
Sep 09 15:50:14.659935 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.659857    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"
Sep 09 15:50:14.660052 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.660032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv" event={"ID":"861de506-45f1-4566-a610-ecf1f980202f","Type":"ContainerDied","Data":"97be43e9ea830385e684e81a0ad0b8e6614c0906a377547f5a041777d5184c7e"}
Sep 09 15:50:14.660122 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.660072    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv" event={"ID":"861de506-45f1-4566-a610-ecf1f980202f","Type":"ContainerDied","Data":"0a5b611e3bb879ca78fc7ba4d623c49b0e5acaf60aa0666ea6218b354ade1ea0"}
Sep 09 15:50:14.660177 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.660127    2578 scope.go:117] "RemoveContainer" containerID="97be43e9ea830385e684e81a0ad0b8e6614c0906a377547f5a041777d5184c7e"
Sep 09 15:50:14.663799 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.663034    2578 generic.go:356] "Generic (PLEG): container finished" podID="dccfee92-8984-45fd-899b-7ab786c9525b" containerID="586338c5d152b213cb89739e85a0237d6f08018f4ffd383f69116d2271242b57" exitCode=0
Sep 09 15:50:14.663799 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.663317    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"
Sep 09 15:50:14.663799 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.663612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2" event={"ID":"dccfee92-8984-45fd-899b-7ab786c9525b","Type":"ContainerDied","Data":"586338c5d152b213cb89739e85a0237d6f08018f4ffd383f69116d2271242b57"}
Sep 09 15:50:14.663799 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.663749    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2" event={"ID":"dccfee92-8984-45fd-899b-7ab786c9525b","Type":"ContainerDied","Data":"c8516e24e9b975917665fdd303dc261060dd680ae3d013600ca289bab8d8e2fc"}
Sep 09 15:50:14.666316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.666287    2578 generic.go:356] "Generic (PLEG): container finished" podID="dae030cb-2e14-41b9-b211-ec0af9f6e9b1" containerID="f7b0e658aa884ddf425515e8019d203b02b9c5a70d5a994989d3ecfc3d6bdf79" exitCode=0
Sep 09 15:50:14.666468 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.666405    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"
Sep 09 15:50:14.666584 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.666516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g" event={"ID":"dae030cb-2e14-41b9-b211-ec0af9f6e9b1","Type":"ContainerDied","Data":"f7b0e658aa884ddf425515e8019d203b02b9c5a70d5a994989d3ecfc3d6bdf79"}
Sep 09 15:50:14.666584 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.666540    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g" event={"ID":"dae030cb-2e14-41b9-b211-ec0af9f6e9b1","Type":"ContainerDied","Data":"7221ec2b063dfd95e14135f54e858387215cceda4a230358d2afeafa3f31b6a0"}
Sep 09 15:50:14.668244 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.668221    2578 generic.go:356] "Generic (PLEG): container finished" podID="97d1bc49-2bac-4edf-a2a8-b953d81fcb9f" containerID="7f6f29ee50f4146173d80fbf12bb3db8e2356664d5b63a9cbee21e07d89ee3c8" exitCode=0
Sep 09 15:50:14.668351 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.668265    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"
Sep 09 15:50:14.668351 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.668313    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d" event={"ID":"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f","Type":"ContainerDied","Data":"7f6f29ee50f4146173d80fbf12bb3db8e2356664d5b63a9cbee21e07d89ee3c8"}
Sep 09 15:50:14.668351 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.668343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d" event={"ID":"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f","Type":"ContainerDied","Data":"14af3faef2899b09aa994ec6ed2773463658781404d1a4e07ff77f9b7111c0c1"}
Sep 09 15:50:14.670141 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.670116    2578 generic.go:356] "Generic (PLEG): container finished" podID="fbf28447-9239-4a6e-adfe-7baa6e38ece4" containerID="2a8a7bc158d79110cfe838879f1a542c9f38e30dd935fbd22238a1d741028cdd" exitCode=0
Sep 09 15:50:14.670244 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.670227    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h" event={"ID":"fbf28447-9239-4a6e-adfe-7baa6e38ece4","Type":"ContainerDied","Data":"2a8a7bc158d79110cfe838879f1a542c9f38e30dd935fbd22238a1d741028cdd"}
Sep 09 15:50:14.670317 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.670253    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h" event={"ID":"fbf28447-9239-4a6e-adfe-7baa6e38ece4","Type":"ContainerDied","Data":"0512da21cfeeddf99ac569dbe67c78ec22e359de20658c79fbd74dfa7e7a412c"}
Sep 09 15:50:14.670317 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.670299    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"
Sep 09 15:50:14.672030 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.671995    2578 generic.go:356] "Generic (PLEG): container finished" podID="6f69e13f-5e3a-4cac-a449-72458cb37ccc" containerID="aa3fcc36cd98b1e1c6d36463eb4ed0214c1539864b4823eb6896c151db028d8c" exitCode=0
Sep 09 15:50:14.672133 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.672075    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg" event={"ID":"6f69e13f-5e3a-4cac-a449-72458cb37ccc","Type":"ContainerDied","Data":"aa3fcc36cd98b1e1c6d36463eb4ed0214c1539864b4823eb6896c151db028d8c"}
Sep 09 15:50:14.672133 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.672095    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"
Sep 09 15:50:14.672133 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.672112    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg" event={"ID":"6f69e13f-5e3a-4cac-a449-72458cb37ccc","Type":"ContainerDied","Data":"edfc55daf686fc72ad55011bd626006189f1551f0d5c62e6e1f684f46dd40e5d"}
Sep 09 15:50:14.672835 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.672817    2578 scope.go:117] "RemoveContainer" containerID="97be43e9ea830385e684e81a0ad0b8e6614c0906a377547f5a041777d5184c7e"
Sep 09 15:50:14.673180 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:50:14.673156    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97be43e9ea830385e684e81a0ad0b8e6614c0906a377547f5a041777d5184c7e\": container with ID starting with 97be43e9ea830385e684e81a0ad0b8e6614c0906a377547f5a041777d5184c7e not found: ID does not exist" containerID="97be43e9ea830385e684e81a0ad0b8e6614c0906a377547f5a041777d5184c7e"
Sep 09 15:50:14.673253 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.673192    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97be43e9ea830385e684e81a0ad0b8e6614c0906a377547f5a041777d5184c7e"} err="failed to get container status \"97be43e9ea830385e684e81a0ad0b8e6614c0906a377547f5a041777d5184c7e\": rpc error: code = NotFound desc = could not find container \"97be43e9ea830385e684e81a0ad0b8e6614c0906a377547f5a041777d5184c7e\": container with ID starting with 97be43e9ea830385e684e81a0ad0b8e6614c0906a377547f5a041777d5184c7e not found: ID does not exist"
Sep 09 15:50:14.673253 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.673215    2578 scope.go:117] "RemoveContainer" containerID="586338c5d152b213cb89739e85a0237d6f08018f4ffd383f69116d2271242b57"
Sep 09 15:50:14.673947 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.673923    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6" event={"ID":"9688bbf2-a894-4955-929e-3d6d6f1726b5","Type":"ContainerStarted","Data":"ac3bd748283fe817ceaddcbb83df1e2bbfa48bd7dc7b9b759a9baaf79963a4e9"}
Sep 09 15:50:14.674117 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.674097    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"
Sep 09 15:50:14.684593 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.684573    2578 scope.go:117] "RemoveContainer" containerID="586338c5d152b213cb89739e85a0237d6f08018f4ffd383f69116d2271242b57"
Sep 09 15:50:14.684905 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:50:14.684884    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"586338c5d152b213cb89739e85a0237d6f08018f4ffd383f69116d2271242b57\": container with ID starting with 586338c5d152b213cb89739e85a0237d6f08018f4ffd383f69116d2271242b57 not found: ID does not exist" containerID="586338c5d152b213cb89739e85a0237d6f08018f4ffd383f69116d2271242b57"
Sep 09 15:50:14.685041 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.684910    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"586338c5d152b213cb89739e85a0237d6f08018f4ffd383f69116d2271242b57"} err="failed to get container status \"586338c5d152b213cb89739e85a0237d6f08018f4ffd383f69116d2271242b57\": rpc error: code = NotFound desc = could not find container \"586338c5d152b213cb89739e85a0237d6f08018f4ffd383f69116d2271242b57\": container with ID starting with 586338c5d152b213cb89739e85a0237d6f08018f4ffd383f69116d2271242b57 not found: ID does not exist"
Sep 09 15:50:14.685041 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.684927    2578 scope.go:117] "RemoveContainer" containerID="f7b0e658aa884ddf425515e8019d203b02b9c5a70d5a994989d3ecfc3d6bdf79"
Sep 09 15:50:14.699640 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.699614    2578 scope.go:117] "RemoveContainer" containerID="f7b0e658aa884ddf425515e8019d203b02b9c5a70d5a994989d3ecfc3d6bdf79"
Sep 09 15:50:14.700698 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:50:14.700662    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b0e658aa884ddf425515e8019d203b02b9c5a70d5a994989d3ecfc3d6bdf79\": container with ID starting with f7b0e658aa884ddf425515e8019d203b02b9c5a70d5a994989d3ecfc3d6bdf79 not found: ID does not exist" containerID="f7b0e658aa884ddf425515e8019d203b02b9c5a70d5a994989d3ecfc3d6bdf79"
Sep 09 15:50:14.700806 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.700706    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b0e658aa884ddf425515e8019d203b02b9c5a70d5a994989d3ecfc3d6bdf79"} err="failed to get container status \"f7b0e658aa884ddf425515e8019d203b02b9c5a70d5a994989d3ecfc3d6bdf79\": rpc error: code = NotFound desc = could not find container \"f7b0e658aa884ddf425515e8019d203b02b9c5a70d5a994989d3ecfc3d6bdf79\": container with ID starting with f7b0e658aa884ddf425515e8019d203b02b9c5a70d5a994989d3ecfc3d6bdf79 not found: ID does not exist"
Sep 09 15:50:14.700806 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.700721    2578 scope.go:117] "RemoveContainer" containerID="7f6f29ee50f4146173d80fbf12bb3db8e2356664d5b63a9cbee21e07d89ee3c8"
Sep 09 15:50:14.702453 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702436    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-dshm\") pod \"861de506-45f1-4566-a610-ecf1f980202f\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") "
Sep 09 15:50:14.702546 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702479    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-model-cache\") pod \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") "
Sep 09 15:50:14.702546 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702508    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-home\") pod \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") "
Sep 09 15:50:14.702546 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702535    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf28447-9239-4a6e-adfe-7baa6e38ece4-tls-certs\") pod \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") "
Sep 09 15:50:14.702712 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702568    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-model-cache\") pod \"861de506-45f1-4566-a610-ecf1f980202f\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") "
Sep 09 15:50:14.702712 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702591    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-dshm\") pod \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") "
Sep 09 15:50:14.702712 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702615    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-tmp-dir\") pod \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") "
Sep 09 15:50:14.702712 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702648    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-model-cache\") pod \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") "
Sep 09 15:50:14.702712 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702671    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-home\") pod \"861de506-45f1-4566-a610-ecf1f980202f\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") "
Sep 09 15:50:14.702712 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702703    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-model-cache\") pod \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") "
Sep 09 15:50:14.703080 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702722    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "97d1bc49-2bac-4edf-a2a8-b953d81fcb9f" (UID: "97d1bc49-2bac-4edf-a2a8-b953d81fcb9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.703080 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702736    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-home\") pod \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") "
Sep 09 15:50:14.703080 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702770    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm46l\" (UniqueName: \"kubernetes.io/projected/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-kube-api-access-zm46l\") pod \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") "
Sep 09 15:50:14.703080 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702803    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-home\") pod \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") "
Sep 09 15:50:14.703080 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702827    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-home\") pod \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") "
Sep 09 15:50:14.703080 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702882    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-tmp-dir\") pod \"861de506-45f1-4566-a610-ecf1f980202f\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") "
Sep 09 15:50:14.703080 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702922    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-dshm\") pod \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") "
Sep 09 15:50:14.703080 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702958    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-model-cache\") pod \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") "
Sep 09 15:50:14.703080 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.702998    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-home" (OuterVolumeSpecName: "home") pod "861de506-45f1-4566-a610-ecf1f980202f" (UID: "861de506-45f1-4566-a610-ecf1f980202f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.703080 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.703012    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n42c2\" (UniqueName: \"kubernetes.io/projected/fbf28447-9239-4a6e-adfe-7baa6e38ece4-kube-api-access-n42c2\") pod \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") "
Sep 09 15:50:14.703080 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.703037    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxl5m\" (UniqueName: \"kubernetes.io/projected/dccfee92-8984-45fd-899b-7ab786c9525b-kube-api-access-rxl5m\") pod \"dccfee92-8984-45fd-899b-7ab786c9525b\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") "
Sep 09 15:50:14.703650 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.703180    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f69e13f-5e3a-4cac-a449-72458cb37ccc" (UID: "6f69e13f-5e3a-4cac-a449-72458cb37ccc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.703650 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.703350    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f69e13f-5e3a-4cac-a449-72458cb37ccc" (UID: "6f69e13f-5e3a-4cac-a449-72458cb37ccc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.703650 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.703515    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "861de506-45f1-4566-a610-ecf1f980202f" (UID: "861de506-45f1-4566-a610-ecf1f980202f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.703812 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.703729    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-model-cache" (OuterVolumeSpecName: "model-cache") pod "fbf28447-9239-4a6e-adfe-7baa6e38ece4" (UID: "fbf28447-9239-4a6e-adfe-7baa6e38ece4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.703912 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.703092    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-dshm\") pod \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") "
Sep 09 15:50:14.704065 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704048    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-tmp-dir\") pod \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") "
Sep 09 15:50:14.704170 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704136    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-home" (OuterVolumeSpecName: "home") pod "dae030cb-2e14-41b9-b211-ec0af9f6e9b1" (UID: "dae030cb-2e14-41b9-b211-ec0af9f6e9b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.704282 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704200    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-dshm\") pod \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\" (UID: \"fbf28447-9239-4a6e-adfe-7baa6e38ece4\") "
Sep 09 15:50:14.704282 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704231    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhgwk\" (UniqueName: \"kubernetes.io/projected/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-kube-api-access-lhgwk\") pod \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") "
Sep 09 15:50:14.704282 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704269    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjmnp\" (UniqueName: \"kubernetes.io/projected/861de506-45f1-4566-a610-ecf1f980202f-kube-api-access-gjmnp\") pod \"861de506-45f1-4566-a610-ecf1f980202f\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") "
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704290    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-dshm\") pod \"dccfee92-8984-45fd-899b-7ab786c9525b\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") "
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704317    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-home\") pod \"dccfee92-8984-45fd-899b-7ab786c9525b\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") "
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704321    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-model-cache" (OuterVolumeSpecName: "model-cache") pod "dae030cb-2e14-41b9-b211-ec0af9f6e9b1" (UID: "dae030cb-2e14-41b9-b211-ec0af9f6e9b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704343    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccfee92-8984-45fd-899b-7ab786c9525b-tls-certs\") pod \"dccfee92-8984-45fd-899b-7ab786c9525b\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") "
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704374    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-tls-certs\") pod \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") "
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704405    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f69e13f-5e3a-4cac-a449-72458cb37ccc-tls-certs\") pod \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") "
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704440    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-tls-certs\") pod \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") "
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704471    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-tmp-dir\") pod \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\" (UID: \"dae030cb-2e14-41b9-b211-ec0af9f6e9b1\") "
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704487    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-home" (OuterVolumeSpecName: "home") pod "6f69e13f-5e3a-4cac-a449-72458cb37ccc" (UID: "6f69e13f-5e3a-4cac-a449-72458cb37ccc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704502    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-tmp-dir\") pod \"dccfee92-8984-45fd-899b-7ab786c9525b\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") "
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704525    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-tmp-dir\") pod \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\" (UID: \"97d1bc49-2bac-4edf-a2a8-b953d81fcb9f\") "
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704556    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpf6w\" (UniqueName: \"kubernetes.io/projected/6f69e13f-5e3a-4cac-a449-72458cb37ccc-kube-api-access-jpf6w\") pod \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\" (UID: \"6f69e13f-5e3a-4cac-a449-72458cb37ccc\") "
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704585    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-model-cache\") pod \"dccfee92-8984-45fd-899b-7ab786c9525b\" (UID: \"dccfee92-8984-45fd-899b-7ab786c9525b\") "
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704614    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/861de506-45f1-4566-a610-ecf1f980202f-tls-certs\") pod \"861de506-45f1-4566-a610-ecf1f980202f\" (UID: \"861de506-45f1-4566-a610-ecf1f980202f\") "
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.704753    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-model-cache" (OuterVolumeSpecName: "model-cache") pod "861de506-45f1-4566-a610-ecf1f980202f" (UID: "861de506-45f1-4566-a610-ecf1f980202f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.705109    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-model-cache\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.705129    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-model-cache\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.705144    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-tmp-dir\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.705157    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-model-cache\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.705263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.705169    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-home\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.706291 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.705181    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-model-cache\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.706291 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.705192    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-home\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.706291 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.705204    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-home\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.706291 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.705215    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-tmp-dir\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.706291 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.705226    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-model-cache\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.706291 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.705384    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-dshm" (OuterVolumeSpecName: "dshm") pod "861de506-45f1-4566-a610-ecf1f980202f" (UID: "861de506-45f1-4566-a610-ecf1f980202f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.706291 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.705447    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-home" (OuterVolumeSpecName: "home") pod "fbf28447-9239-4a6e-adfe-7baa6e38ece4" (UID: "fbf28447-9239-4a6e-adfe-7baa6e38ece4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.706291 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.705620    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-home" (OuterVolumeSpecName: "home") pod "97d1bc49-2bac-4edf-a2a8-b953d81fcb9f" (UID: "97d1bc49-2bac-4edf-a2a8-b953d81fcb9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.706685 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.706308    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-dshm" (OuterVolumeSpecName: "dshm") pod "97d1bc49-2bac-4edf-a2a8-b953d81fcb9f" (UID: "97d1bc49-2bac-4edf-a2a8-b953d81fcb9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.706685 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.706669    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dae030cb-2e14-41b9-b211-ec0af9f6e9b1" (UID: "dae030cb-2e14-41b9-b211-ec0af9f6e9b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.709321 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.709293    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-home" (OuterVolumeSpecName: "home") pod "dccfee92-8984-45fd-899b-7ab786c9525b" (UID: "dccfee92-8984-45fd-899b-7ab786c9525b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.709532 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.709496    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97d1bc49-2bac-4edf-a2a8-b953d81fcb9f" (UID: "97d1bc49-2bac-4edf-a2a8-b953d81fcb9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.709733 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.709713    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dccfee92-8984-45fd-899b-7ab786c9525b" (UID: "dccfee92-8984-45fd-899b-7ab786c9525b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.712708 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.712682    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dae030cb-2e14-41b9-b211-ec0af9f6e9b1" (UID: "dae030cb-2e14-41b9-b211-ec0af9f6e9b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:50:14.712927 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.712902    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/861de506-45f1-4566-a610-ecf1f980202f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "861de506-45f1-4566-a610-ecf1f980202f" (UID: "861de506-45f1-4566-a610-ecf1f980202f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:50:14.713141 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.713112    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-dshm" (OuterVolumeSpecName: "dshm") pod "dccfee92-8984-45fd-899b-7ab786c9525b" (UID: "dccfee92-8984-45fd-899b-7ab786c9525b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.713292 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.713272    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-model-cache" (OuterVolumeSpecName: "model-cache") pod "dccfee92-8984-45fd-899b-7ab786c9525b" (UID: "dccfee92-8984-45fd-899b-7ab786c9525b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.713604 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.713523    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf28447-9239-4a6e-adfe-7baa6e38ece4-kube-api-access-n42c2" (OuterVolumeSpecName: "kube-api-access-n42c2") pod "fbf28447-9239-4a6e-adfe-7baa6e38ece4" (UID: "fbf28447-9239-4a6e-adfe-7baa6e38ece4"). InnerVolumeSpecName "kube-api-access-n42c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:50:14.713604 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.713560    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f69e13f-5e3a-4cac-a449-72458cb37ccc-kube-api-access-jpf6w" (OuterVolumeSpecName: "kube-api-access-jpf6w") pod "6f69e13f-5e3a-4cac-a449-72458cb37ccc" (UID: "6f69e13f-5e3a-4cac-a449-72458cb37ccc"). InnerVolumeSpecName "kube-api-access-jpf6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:50:14.713604 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.713572    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-dshm" (OuterVolumeSpecName: "dshm") pod "fbf28447-9239-4a6e-adfe-7baa6e38ece4" (UID: "fbf28447-9239-4a6e-adfe-7baa6e38ece4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.713809 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.713612    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-dshm" (OuterVolumeSpecName: "dshm") pod "6f69e13f-5e3a-4cac-a449-72458cb37ccc" (UID: "6f69e13f-5e3a-4cac-a449-72458cb37ccc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.713809 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.713666    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-kube-api-access-zm46l" (OuterVolumeSpecName: "kube-api-access-zm46l") pod "97d1bc49-2bac-4edf-a2a8-b953d81fcb9f" (UID: "97d1bc49-2bac-4edf-a2a8-b953d81fcb9f"). InnerVolumeSpecName "kube-api-access-zm46l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:50:14.713932 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.713806    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-dshm" (OuterVolumeSpecName: "dshm") pod "dae030cb-2e14-41b9-b211-ec0af9f6e9b1" (UID: "dae030cb-2e14-41b9-b211-ec0af9f6e9b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.713932 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.713829    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861de506-45f1-4566-a610-ecf1f980202f-kube-api-access-gjmnp" (OuterVolumeSpecName: "kube-api-access-gjmnp") pod "861de506-45f1-4566-a610-ecf1f980202f" (UID: "861de506-45f1-4566-a610-ecf1f980202f"). InnerVolumeSpecName "kube-api-access-gjmnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:50:14.713932 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.713912    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dccfee92-8984-45fd-899b-7ab786c9525b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dccfee92-8984-45fd-899b-7ab786c9525b" (UID: "dccfee92-8984-45fd-899b-7ab786c9525b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:50:14.714778 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.714164    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbf28447-9239-4a6e-adfe-7baa6e38ece4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fbf28447-9239-4a6e-adfe-7baa6e38ece4" (UID: "fbf28447-9239-4a6e-adfe-7baa6e38ece4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:50:14.714877 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.714784    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fbf28447-9239-4a6e-adfe-7baa6e38ece4" (UID: "fbf28447-9239-4a6e-adfe-7baa6e38ece4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:50:14.715384 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.715351    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-kube-api-access-lhgwk" (OuterVolumeSpecName: "kube-api-access-lhgwk") pod "dae030cb-2e14-41b9-b211-ec0af9f6e9b1" (UID: "dae030cb-2e14-41b9-b211-ec0af9f6e9b1"). InnerVolumeSpecName "kube-api-access-lhgwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:50:14.715819 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.715645    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6" podStartSLOduration=2.291869981 podStartE2EDuration="2.715627679s" totalImagesPullingTime="423.757698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:50:14.714632953 +0000 UTC m=+1804.356674942" watchObservedRunningTime="2026-09-09 15:50:14.715627679 +0000 UTC m=+1804.357669669"
Sep 09 15:50:14.715819 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.715705    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f69e13f-5e3a-4cac-a449-72458cb37ccc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f69e13f-5e3a-4cac-a449-72458cb37ccc" (UID: "6f69e13f-5e3a-4cac-a449-72458cb37ccc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:50:14.716379 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.716349    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dccfee92-8984-45fd-899b-7ab786c9525b-kube-api-access-rxl5m" (OuterVolumeSpecName: "kube-api-access-rxl5m") pod "dccfee92-8984-45fd-899b-7ab786c9525b" (UID: "dccfee92-8984-45fd-899b-7ab786c9525b"). InnerVolumeSpecName "kube-api-access-rxl5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:50:14.717496 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.717456    2578 scope.go:117] "RemoveContainer" containerID="7f6f29ee50f4146173d80fbf12bb3db8e2356664d5b63a9cbee21e07d89ee3c8"
Sep 09 15:50:14.717587 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.717457    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97d1bc49-2bac-4edf-a2a8-b953d81fcb9f" (UID: "97d1bc49-2bac-4edf-a2a8-b953d81fcb9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:50:14.717819 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:50:14.717790    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f6f29ee50f4146173d80fbf12bb3db8e2356664d5b63a9cbee21e07d89ee3c8\": container with ID starting with 7f6f29ee50f4146173d80fbf12bb3db8e2356664d5b63a9cbee21e07d89ee3c8 not found: ID does not exist" containerID="7f6f29ee50f4146173d80fbf12bb3db8e2356664d5b63a9cbee21e07d89ee3c8"
Sep 09 15:50:14.717923 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.717828    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f6f29ee50f4146173d80fbf12bb3db8e2356664d5b63a9cbee21e07d89ee3c8"} err="failed to get container status \"7f6f29ee50f4146173d80fbf12bb3db8e2356664d5b63a9cbee21e07d89ee3c8\": rpc error: code = NotFound desc = could not find container \"7f6f29ee50f4146173d80fbf12bb3db8e2356664d5b63a9cbee21e07d89ee3c8\": container with ID starting with 7f6f29ee50f4146173d80fbf12bb3db8e2356664d5b63a9cbee21e07d89ee3c8 not found: ID does not exist"
Sep 09 15:50:14.717923 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.717849    2578 scope.go:117] "RemoveContainer" containerID="2a8a7bc158d79110cfe838879f1a542c9f38e30dd935fbd22238a1d741028cdd"
Sep 09 15:50:14.727458 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.727442    2578 scope.go:117] "RemoveContainer" containerID="2a8a7bc158d79110cfe838879f1a542c9f38e30dd935fbd22238a1d741028cdd"
Sep 09 15:50:14.727777 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:50:14.727759    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a8a7bc158d79110cfe838879f1a542c9f38e30dd935fbd22238a1d741028cdd\": container with ID starting with 2a8a7bc158d79110cfe838879f1a542c9f38e30dd935fbd22238a1d741028cdd not found: ID does not exist" containerID="2a8a7bc158d79110cfe838879f1a542c9f38e30dd935fbd22238a1d741028cdd"
Sep 09 15:50:14.727864 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.727790    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8a7bc158d79110cfe838879f1a542c9f38e30dd935fbd22238a1d741028cdd"} err="failed to get container status \"2a8a7bc158d79110cfe838879f1a542c9f38e30dd935fbd22238a1d741028cdd\": rpc error: code = NotFound desc = could not find container \"2a8a7bc158d79110cfe838879f1a542c9f38e30dd935fbd22238a1d741028cdd\": container with ID starting with 2a8a7bc158d79110cfe838879f1a542c9f38e30dd935fbd22238a1d741028cdd not found: ID does not exist"
Sep 09 15:50:14.727864 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.727813    2578 scope.go:117] "RemoveContainer" containerID="aa3fcc36cd98b1e1c6d36463eb4ed0214c1539864b4823eb6896c151db028d8c"
Sep 09 15:50:14.735777 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.735758    2578 scope.go:117] "RemoveContainer" containerID="aa3fcc36cd98b1e1c6d36463eb4ed0214c1539864b4823eb6896c151db028d8c"
Sep 09 15:50:14.736195 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:50:14.736141    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa3fcc36cd98b1e1c6d36463eb4ed0214c1539864b4823eb6896c151db028d8c\": container with ID starting with aa3fcc36cd98b1e1c6d36463eb4ed0214c1539864b4823eb6896c151db028d8c not found: ID does not exist" containerID="aa3fcc36cd98b1e1c6d36463eb4ed0214c1539864b4823eb6896c151db028d8c"
Sep 09 15:50:14.736195 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.736176    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3fcc36cd98b1e1c6d36463eb4ed0214c1539864b4823eb6896c151db028d8c"} err="failed to get container status \"aa3fcc36cd98b1e1c6d36463eb4ed0214c1539864b4823eb6896c151db028d8c\": rpc error: code = NotFound desc = could not find container \"aa3fcc36cd98b1e1c6d36463eb4ed0214c1539864b4823eb6896c151db028d8c\": container with ID starting with aa3fcc36cd98b1e1c6d36463eb4ed0214c1539864b4823eb6896c151db028d8c not found: ID does not exist"
Sep 09 15:50:14.806507 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806472    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-tmp-dir\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806507 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806505    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-tmp-dir\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806518    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-tmp-dir\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806531    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpf6w\" (UniqueName: \"kubernetes.io/projected/6f69e13f-5e3a-4cac-a449-72458cb37ccc-kube-api-access-jpf6w\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806542    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-model-cache\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806554    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/861de506-45f1-4566-a610-ecf1f980202f-tls-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806565    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/861de506-45f1-4566-a610-ecf1f980202f-dshm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806578    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-home\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806590    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf28447-9239-4a6e-adfe-7baa6e38ece4-tls-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806603    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-dshm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806616    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm46l\" (UniqueName: \"kubernetes.io/projected/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-kube-api-access-zm46l\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806628    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-home\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806640    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f69e13f-5e3a-4cac-a449-72458cb37ccc-dshm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806654    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n42c2\" (UniqueName: \"kubernetes.io/projected/fbf28447-9239-4a6e-adfe-7baa6e38ece4-kube-api-access-n42c2\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806669    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxl5m\" (UniqueName: \"kubernetes.io/projected/dccfee92-8984-45fd-899b-7ab786c9525b-kube-api-access-rxl5m\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806682    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-dshm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806694    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-tmp-dir\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806704    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf28447-9239-4a6e-adfe-7baa6e38ece4-dshm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806715    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhgwk\" (UniqueName: \"kubernetes.io/projected/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-kube-api-access-lhgwk\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806724    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjmnp\" (UniqueName: \"kubernetes.io/projected/861de506-45f1-4566-a610-ecf1f980202f-kube-api-access-gjmnp\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806732    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-dshm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806740    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccfee92-8984-45fd-899b-7ab786c9525b-home\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806747    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccfee92-8984-45fd-899b-7ab786c9525b-tls-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806754    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae030cb-2e14-41b9-b211-ec0af9f6e9b1-tls-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806761    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f69e13f-5e3a-4cac-a449-72458cb37ccc-tls-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.806753 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.806768    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f-tls-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:50:14.942826 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:50:14.942798    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97d1bc49_2bac_4edf_a2a8_b953d81fcb9f.slice/crio-14af3faef2899b09aa994ec6ed2773463658781404d1a4e07ff77f9b7111c0c1\": RecentStats: unable to find data in memory cache]"
Sep 09 15:50:14.975013 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.974966    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"]
Sep 09 15:50:14.978307 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.978287    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv"]
Sep 09 15:50:14.987712 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.987649    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"]
Sep 09 15:50:14.989764 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.989746    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/unconfigured-w0-df30be-kserve-55bf795944-r7j4d"]
Sep 09 15:50:14.998542 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:14.998524    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"]
Sep 09 15:50:15.002645 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:15.002624    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/embedding-w0-df30be-kserve-66b6757b4d-wbh5g"]
Sep 09 15:50:15.013851 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:15.013829    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"]
Sep 09 15:50:15.017553 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:15.017534    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/simulator-w0-df30be-kserve-cdc86ff49-7zb4h"]
Sep 09 15:50:15.027196 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:15.027175    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"]
Sep 09 15:50:15.031278 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:15.031259    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/distinct-w0-df30be-kserve-88c5dcd79-qrhq2"]
Sep 09 15:50:15.041417 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:15.041394    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"]
Sep 09 15:50:15.047391 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:15.047368    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-df30be/distinct-2-w0-df30be-kserve-67c489f8d8-tgdmg"]
Sep 09 15:50:15.214027 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:15.214004    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 15:50:15.219469 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:15.219448    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 15:50:15.221719 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:15.221702    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:50:15.226633 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:15.226616    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:50:16.867056 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:16.867025    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f69e13f-5e3a-4cac-a449-72458cb37ccc" path="/var/lib/kubelet/pods/6f69e13f-5e3a-4cac-a449-72458cb37ccc/volumes"
Sep 09 15:50:16.867450 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:16.867435    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="861de506-45f1-4566-a610-ecf1f980202f" path="/var/lib/kubelet/pods/861de506-45f1-4566-a610-ecf1f980202f/volumes"
Sep 09 15:50:16.867795 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:16.867783    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97d1bc49-2bac-4edf-a2a8-b953d81fcb9f" path="/var/lib/kubelet/pods/97d1bc49-2bac-4edf-a2a8-b953d81fcb9f/volumes"
Sep 09 15:50:16.868476 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:16.868460    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dae030cb-2e14-41b9-b211-ec0af9f6e9b1" path="/var/lib/kubelet/pods/dae030cb-2e14-41b9-b211-ec0af9f6e9b1/volumes"
Sep 09 15:50:16.868843 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:16.868831    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dccfee92-8984-45fd-899b-7ab786c9525b" path="/var/lib/kubelet/pods/dccfee92-8984-45fd-899b-7ab786c9525b/volumes"
Sep 09 15:50:16.869222 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:16.869205    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbf28447-9239-4a6e-adfe-7baa6e38ece4" path="/var/lib/kubelet/pods/fbf28447-9239-4a6e-adfe-7baa6e38ece4/volumes"
Sep 09 15:50:17.834687 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:17.834652    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:17.906184 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:17.906156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:18.144025 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:18.143990    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:19.528845 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:19.528739    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df30be/premium-w0-df30be-kserve-6d79b86fc6-hpzmv" podUID="861de506-45f1-4566-a610-ecf1f980202f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 09 15:50:20.690305 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:20.690279    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"
Sep 09 15:50:25.140583 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:25.140544    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:25.240556 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:25.240519    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:35.802375 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:35.802326    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:35.901252 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:35.901214    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:36.625674 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:36.625615    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:36.685468 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:36.685433    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:37.263910 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:37.263868    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:37.690294 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:37.690254    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:38.064698 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:38.064618    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:41.557197 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:41.557164    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:41.655394 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:41.655359    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:42.657343 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657312    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"]
Sep 09 15:50:42.657766 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657753    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dccfee92-8984-45fd-899b-7ab786c9525b" containerName="main"
Sep 09 15:50:42.657814 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657769    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dccfee92-8984-45fd-899b-7ab786c9525b" containerName="main"
Sep 09 15:50:42.657814 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657778    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97d1bc49-2bac-4edf-a2a8-b953d81fcb9f" containerName="main"
Sep 09 15:50:42.657814 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657784    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="97d1bc49-2bac-4edf-a2a8-b953d81fcb9f" containerName="main"
Sep 09 15:50:42.657814 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657803    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="861de506-45f1-4566-a610-ecf1f980202f" containerName="main"
Sep 09 15:50:42.657814 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657808    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="861de506-45f1-4566-a610-ecf1f980202f" containerName="main"
Sep 09 15:50:42.657971 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657817    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbf28447-9239-4a6e-adfe-7baa6e38ece4" containerName="main"
Sep 09 15:50:42.657971 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657822    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbf28447-9239-4a6e-adfe-7baa6e38ece4" containerName="main"
Sep 09 15:50:42.657971 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657835    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dae030cb-2e14-41b9-b211-ec0af9f6e9b1" containerName="main"
Sep 09 15:50:42.657971 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657840    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dae030cb-2e14-41b9-b211-ec0af9f6e9b1" containerName="main"
Sep 09 15:50:42.657971 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657848    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f69e13f-5e3a-4cac-a449-72458cb37ccc" containerName="main"
Sep 09 15:50:42.657971 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657854    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f69e13f-5e3a-4cac-a449-72458cb37ccc" containerName="main"
Sep 09 15:50:42.657971 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657915    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dae030cb-2e14-41b9-b211-ec0af9f6e9b1" containerName="main"
Sep 09 15:50:42.657971 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657924    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dccfee92-8984-45fd-899b-7ab786c9525b" containerName="main"
Sep 09 15:50:42.657971 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657931    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="861de506-45f1-4566-a610-ecf1f980202f" containerName="main"
Sep 09 15:50:42.657971 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657937    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="97d1bc49-2bac-4edf-a2a8-b953d81fcb9f" containerName="main"
Sep 09 15:50:42.657971 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657944    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f69e13f-5e3a-4cac-a449-72458cb37ccc" containerName="main"
Sep 09 15:50:42.657971 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.657950    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbf28447-9239-4a6e-adfe-7baa6e38ece4" containerName="main"
Sep 09 15:50:42.661061 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.661015    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"
Sep 09 15:50:42.663058 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.663036    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ce9689-serving-cert\""
Sep 09 15:50:42.672782 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.672757    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"]
Sep 09 15:50:42.689850 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.689827    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:42.785595 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.785556    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b776c293-c476-45b8-ac53-ef6aef8cb13f-metrics-certs\") pod \"maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs\" (UID: \"b776c293-c476-45b8-ac53-ef6aef8cb13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"
Sep 09 15:50:42.785788 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.785603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg5s6\" (UniqueName: \"kubernetes.io/projected/b776c293-c476-45b8-ac53-ef6aef8cb13f-kube-api-access-wg5s6\") pod \"maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs\" (UID: \"b776c293-c476-45b8-ac53-ef6aef8cb13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"
Sep 09 15:50:42.785788 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.785696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b776c293-c476-45b8-ac53-ef6aef8cb13f-openshift-service-ca\") pod \"maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs\" (UID: \"b776c293-c476-45b8-ac53-ef6aef8cb13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"
Sep 09 15:50:42.785788 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.785747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b776c293-c476-45b8-ac53-ef6aef8cb13f-maas-api-tls\") pod \"maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs\" (UID: \"b776c293-c476-45b8-ac53-ef6aef8cb13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"
Sep 09 15:50:42.790755 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.790724    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:42.886467 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.886438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b776c293-c476-45b8-ac53-ef6aef8cb13f-maas-api-tls\") pod \"maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs\" (UID: \"b776c293-c476-45b8-ac53-ef6aef8cb13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"
Sep 09 15:50:42.886652 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.886510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b776c293-c476-45b8-ac53-ef6aef8cb13f-metrics-certs\") pod \"maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs\" (UID: \"b776c293-c476-45b8-ac53-ef6aef8cb13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"
Sep 09 15:50:42.886652 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.886544    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg5s6\" (UniqueName: \"kubernetes.io/projected/b776c293-c476-45b8-ac53-ef6aef8cb13f-kube-api-access-wg5s6\") pod \"maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs\" (UID: \"b776c293-c476-45b8-ac53-ef6aef8cb13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"
Sep 09 15:50:42.886652 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.886584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b776c293-c476-45b8-ac53-ef6aef8cb13f-openshift-service-ca\") pod \"maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs\" (UID: \"b776c293-c476-45b8-ac53-ef6aef8cb13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"
Sep 09 15:50:42.887266 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.887246    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b776c293-c476-45b8-ac53-ef6aef8cb13f-openshift-service-ca\") pod \"maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs\" (UID: \"b776c293-c476-45b8-ac53-ef6aef8cb13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"
Sep 09 15:50:42.888941 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.888916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b776c293-c476-45b8-ac53-ef6aef8cb13f-maas-api-tls\") pod \"maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs\" (UID: \"b776c293-c476-45b8-ac53-ef6aef8cb13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"
Sep 09 15:50:42.889296 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.889281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b776c293-c476-45b8-ac53-ef6aef8cb13f-metrics-certs\") pod \"maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs\" (UID: \"b776c293-c476-45b8-ac53-ef6aef8cb13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"
Sep 09 15:50:42.899237 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.899217    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg5s6\" (UniqueName: \"kubernetes.io/projected/b776c293-c476-45b8-ac53-ef6aef8cb13f-kube-api-access-wg5s6\") pod \"maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs\" (UID: \"b776c293-c476-45b8-ac53-ef6aef8cb13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"
Sep 09 15:50:42.909521 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.909473    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:42.974489 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:42.974459    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"
Sep 09 15:50:43.252822 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:43.252793    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"]
Sep 09 15:50:43.254631 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:50:43.254587    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb776c293_c476_45b8_ac53_ef6aef8cb13f.slice/crio-cb06c97179467429ad50ec12b3152009b4ccd5be50aba5df3b944dc8bbc0bfad WatchSource:0}: Error finding container cb06c97179467429ad50ec12b3152009b4ccd5be50aba5df3b944dc8bbc0bfad: Status 404 returned error can't find the container with id cb06c97179467429ad50ec12b3152009b4ccd5be50aba5df3b944dc8bbc0bfad
Sep 09 15:50:43.281726 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:43.281703    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:43.376496 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:43.376458    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:50:43.846070 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:43.846030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs" event={"ID":"b776c293-c476-45b8-ac53-ef6aef8cb13f","Type":"ContainerStarted","Data":"1b14354122e5f564e503ce6fbb6354d0aed97ef1e12350ed19e422224ef61d44"}
Sep 09 15:50:43.846492 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:43.846094    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs" event={"ID":"b776c293-c476-45b8-ac53-ef6aef8cb13f","Type":"ContainerStarted","Data":"cb06c97179467429ad50ec12b3152009b4ccd5be50aba5df3b944dc8bbc0bfad"}
Sep 09 15:50:43.846492 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:43.846154    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"
Sep 09 15:50:43.862760 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:43.862709    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs" podStartSLOduration=1.502164141 podStartE2EDuration="1.862694315s" totalImagesPullingTime="360.530174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:50:43.860548875 +0000 UTC m=+1833.502590865" watchObservedRunningTime="2026-09-09 15:50:43.862694315 +0000 UTC m=+1833.504736303"
Sep 09 15:50:49.858872 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:50:49.858833    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"
Sep 09 15:51:05.239028 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.238990    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"]
Sep 09 15:51:05.242649 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.242632    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.245121 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.245096    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1e15c8\"/\"e2e-conflict-a-ef131f3d-kserve-self-signed-certs\""
Sep 09 15:51:05.250340 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.250317    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"]
Sep 09 15:51:05.320574 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.320532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-dshm\") pod \"e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.320765 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.320598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343e2e2d-46c8-4f28-a600-663f2e06c467-tls-certs\") pod \"e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.320765 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.320664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvlcn\" (UniqueName: \"kubernetes.io/projected/343e2e2d-46c8-4f28-a600-663f2e06c467-kube-api-access-dvlcn\") pod \"e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.320765 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.320737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-home\") pod \"e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.320900 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.320779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-model-cache\") pod \"e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.320900 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.320808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-tmp-dir\") pod \"e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.421904 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.421862    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-home\") pod \"e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.422132 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.421910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-model-cache\") pod \"e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.422132 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.421944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-tmp-dir\") pod \"e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.422132 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.422046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-dshm\") pod \"e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.422306 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.422130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343e2e2d-46c8-4f28-a600-663f2e06c467-tls-certs\") pod \"e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.422306 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.422202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvlcn\" (UniqueName: \"kubernetes.io/projected/343e2e2d-46c8-4f28-a600-663f2e06c467-kube-api-access-dvlcn\") pod \"e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.422478 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.422451    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-home\") pod \"e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.422552 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.422521    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-model-cache\") pod \"e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.422552 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.422535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-tmp-dir\") pod \"e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.424487 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.424467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-dshm\") pod \"e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.424896 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.424873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343e2e2d-46c8-4f28-a600-663f2e06c467-tls-certs\") pod \"e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.429702 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.429664    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvlcn\" (UniqueName: \"kubernetes.io/projected/343e2e2d-46c8-4f28-a600-663f2e06c467-kube-api-access-dvlcn\") pod \"e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.542494 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.542410    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:05.554263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.554228    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:05.825834 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.825802    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"]
Sep 09 15:51:05.828453 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:51:05.828424    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod343e2e2d_46c8_4f28_a600_663f2e06c467.slice/crio-0f4cc886076d1ded85c95713c8d6adc185b19e4ad5a860c275e3eaedce1c42bf WatchSource:0}: Error finding container 0f4cc886076d1ded85c95713c8d6adc185b19e4ad5a860c275e3eaedce1c42bf: Status 404 returned error can't find the container with id 0f4cc886076d1ded85c95713c8d6adc185b19e4ad5a860c275e3eaedce1c42bf
Sep 09 15:51:05.969042 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.969005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf" event={"ID":"343e2e2d-46c8-4f28-a600-663f2e06c467","Type":"ContainerStarted","Data":"f608140366ff97532c3c6b6b3fc8c1d5bb5f9375322da5f04fe9332c880c047d"}
Sep 09 15:51:05.969042 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.969050    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf" event={"ID":"343e2e2d-46c8-4f28-a600-663f2e06c467","Type":"ContainerStarted","Data":"0f4cc886076d1ded85c95713c8d6adc185b19e4ad5a860c275e3eaedce1c42bf"}
Sep 09 15:51:05.989141 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:05.989081    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf" podStartSLOduration=0.989060666 podStartE2EDuration="989.060666ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:05.984401311 +0000 UTC m=+1855.626443313" watchObservedRunningTime="2026-09-09 15:51:05.989060666 +0000 UTC m=+1855.631102655"
Sep 09 15:51:06.826463 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:06.826429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:10.989125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:10.989097    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:15.555220 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:15.555180    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:15.555623 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:15.555237    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:15.567953 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:15.567928    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:16.032194 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:16.032159    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:51:17.226472 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:17.226440    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:26.260553 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.260518    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"]
Sep 09 15:51:26.261394 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.261335    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6" podUID="9688bbf2-a894-4955-929e-3d6d6f1726b5" containerName="maas-api" containerID="cri-o://ac3bd748283fe817ceaddcbb83df1e2bbfa48bd7dc7b9b759a9baaf79963a4e9" gracePeriod=30
Sep 09 15:51:26.295191 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.295161    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:26.373636 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.373608    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:26.439183 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.439144    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:26.504225 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.504196    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:26.510232 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.510213    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"
Sep 09 15:51:26.643060 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.643028    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9688bbf2-a894-4955-929e-3d6d6f1726b5-metrics-certs\") pod \"9688bbf2-a894-4955-929e-3d6d6f1726b5\" (UID: \"9688bbf2-a894-4955-929e-3d6d6f1726b5\") "
Sep 09 15:51:26.643371 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.643117    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9688bbf2-a894-4955-929e-3d6d6f1726b5-maas-api-tls\") pod \"9688bbf2-a894-4955-929e-3d6d6f1726b5\" (UID: \"9688bbf2-a894-4955-929e-3d6d6f1726b5\") "
Sep 09 15:51:26.643371 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.643195    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9688bbf2-a894-4955-929e-3d6d6f1726b5-openshift-service-ca\") pod \"9688bbf2-a894-4955-929e-3d6d6f1726b5\" (UID: \"9688bbf2-a894-4955-929e-3d6d6f1726b5\") "
Sep 09 15:51:26.643371 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.643220    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwmgd\" (UniqueName: \"kubernetes.io/projected/9688bbf2-a894-4955-929e-3d6d6f1726b5-kube-api-access-zwmgd\") pod \"9688bbf2-a894-4955-929e-3d6d6f1726b5\" (UID: \"9688bbf2-a894-4955-929e-3d6d6f1726b5\") "
Sep 09 15:51:26.643550 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.643529    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9688bbf2-a894-4955-929e-3d6d6f1726b5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9688bbf2-a894-4955-929e-3d6d6f1726b5" (UID: "9688bbf2-a894-4955-929e-3d6d6f1726b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:51:26.645496 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.645468    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9688bbf2-a894-4955-929e-3d6d6f1726b5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9688bbf2-a894-4955-929e-3d6d6f1726b5" (UID: "9688bbf2-a894-4955-929e-3d6d6f1726b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:51:26.645698 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.645670    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9688bbf2-a894-4955-929e-3d6d6f1726b5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9688bbf2-a894-4955-929e-3d6d6f1726b5" (UID: "9688bbf2-a894-4955-929e-3d6d6f1726b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:51:26.645698 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.645672    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9688bbf2-a894-4955-929e-3d6d6f1726b5-kube-api-access-zwmgd" (OuterVolumeSpecName: "kube-api-access-zwmgd") pod "9688bbf2-a894-4955-929e-3d6d6f1726b5" (UID: "9688bbf2-a894-4955-929e-3d6d6f1726b5"). InnerVolumeSpecName "kube-api-access-zwmgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:51:26.744878 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.744837    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9688bbf2-a894-4955-929e-3d6d6f1726b5-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:51:26.744878 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.744869    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9688bbf2-a894-4955-929e-3d6d6f1726b5-maas-api-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:51:26.744878 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.744879    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9688bbf2-a894-4955-929e-3d6d6f1726b5-openshift-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:51:26.745143 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.744889    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwmgd\" (UniqueName: \"kubernetes.io/projected/9688bbf2-a894-4955-929e-3d6d6f1726b5-kube-api-access-zwmgd\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:51:26.951651 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:26.951578    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:27.079862 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:27.079828    2578 generic.go:356] "Generic (PLEG): container finished" podID="9688bbf2-a894-4955-929e-3d6d6f1726b5" containerID="ac3bd748283fe817ceaddcbb83df1e2bbfa48bd7dc7b9b759a9baaf79963a4e9" exitCode=0
Sep 09 15:51:27.080075 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:27.079868    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"
Sep 09 15:51:27.080075 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:27.079902    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6" event={"ID":"9688bbf2-a894-4955-929e-3d6d6f1726b5","Type":"ContainerDied","Data":"ac3bd748283fe817ceaddcbb83df1e2bbfa48bd7dc7b9b759a9baaf79963a4e9"}
Sep 09 15:51:27.080075 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:27.079953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6" event={"ID":"9688bbf2-a894-4955-929e-3d6d6f1726b5","Type":"ContainerDied","Data":"ca30022b3ed8fcbbc2d0e76c950181024d58ea9149f9f06dc38684908988cb6a"}
Sep 09 15:51:27.080075 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:27.080000    2578 scope.go:117] "RemoveContainer" containerID="ac3bd748283fe817ceaddcbb83df1e2bbfa48bd7dc7b9b759a9baaf79963a4e9"
Sep 09 15:51:27.088942 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:27.088923    2578 scope.go:117] "RemoveContainer" containerID="ac3bd748283fe817ceaddcbb83df1e2bbfa48bd7dc7b9b759a9baaf79963a4e9"
Sep 09 15:51:27.089235 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:51:27.089217    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3bd748283fe817ceaddcbb83df1e2bbfa48bd7dc7b9b759a9baaf79963a4e9\": container with ID starting with ac3bd748283fe817ceaddcbb83df1e2bbfa48bd7dc7b9b759a9baaf79963a4e9 not found: ID does not exist" containerID="ac3bd748283fe817ceaddcbb83df1e2bbfa48bd7dc7b9b759a9baaf79963a4e9"
Sep 09 15:51:27.089286 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:27.089248    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3bd748283fe817ceaddcbb83df1e2bbfa48bd7dc7b9b759a9baaf79963a4e9"} err="failed to get container status \"ac3bd748283fe817ceaddcbb83df1e2bbfa48bd7dc7b9b759a9baaf79963a4e9\": rpc error: code = NotFound desc = could not find container \"ac3bd748283fe817ceaddcbb83df1e2bbfa48bd7dc7b9b759a9baaf79963a4e9\": container with ID starting with ac3bd748283fe817ceaddcbb83df1e2bbfa48bd7dc7b9b759a9baaf79963a4e9 not found: ID does not exist"
Sep 09 15:51:27.106156 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:27.106131    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"]
Sep 09 15:51:27.116274 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:27.116253    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3ac2fa-77c988d7b5-6nrh6"]
Sep 09 15:51:28.867440 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:28.867408    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9688bbf2-a894-4955-929e-3d6d6f1726b5" path="/var/lib/kubelet/pods/9688bbf2-a894-4955-929e-3d6d6f1726b5/volumes"
Sep 09 15:51:32.621799 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:32.621722    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:32.721615 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:32.721585    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:32.935218 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:32.935188    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:33.377865 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:33.377789    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:47.550851 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.550814    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"]
Sep 09 15:51:47.551342 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.551257    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9688bbf2-a894-4955-929e-3d6d6f1726b5" containerName="maas-api"
Sep 09 15:51:47.551342 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.551269    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9688bbf2-a894-4955-929e-3d6d6f1726b5" containerName="maas-api"
Sep 09 15:51:47.551419 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.551351    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9688bbf2-a894-4955-929e-3d6d6f1726b5" containerName="maas-api"
Sep 09 15:51:47.554196 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.554171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.556706 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.556681    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1e15c8\"/\"e2e-conflict-b-ef131f3d-kserve-self-signed-certs\""
Sep 09 15:51:47.565032 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.564971    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"]
Sep 09 15:51:47.644253 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.644218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-tmp-dir\") pod \"e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.644429 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.644264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-dshm\") pod \"e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.644429 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.644353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqgbg\" (UniqueName: \"kubernetes.io/projected/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-kube-api-access-cqgbg\") pod \"e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.644529 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.644445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-model-cache\") pod \"e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.644529 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.644491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-tls-certs\") pod \"e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.644529 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.644512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-home\") pod \"e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.660588 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.660561    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:47.745882 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.745853    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-model-cache\") pod \"e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.746088 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.745904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-tls-certs\") pod \"e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.746173 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.746109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-home\") pod \"e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.746240 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.746192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-tmp-dir\") pod \"e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.746240 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.746222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-dshm\") pod \"e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.746348 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.746292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqgbg\" (UniqueName: \"kubernetes.io/projected/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-kube-api-access-cqgbg\") pod \"e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.746348 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.746301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-model-cache\") pod \"e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.746530 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.746509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-home\") pod \"e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.746619 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.746525    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-tmp-dir\") pod \"e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.748530 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.748504    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-tls-certs\") pod \"e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.748800 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.748782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-dshm\") pod \"e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.755240 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.755207    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqgbg\" (UniqueName: \"kubernetes.io/projected/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-kube-api-access-cqgbg\") pod \"e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") " pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:47.868179 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:47.868088    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:48.142387 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:48.142359    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"]
Sep 09 15:51:48.144534 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:51:48.144506    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65a4a882_9e17_4dfb_8a3d_45eabcf6bd7d.slice/crio-8f630cf4b3f50b875bfcfe838267523b3c625451467f39edf2c5d1b5c1f85b38 WatchSource:0}: Error finding container 8f630cf4b3f50b875bfcfe838267523b3c625451467f39edf2c5d1b5c1f85b38: Status 404 returned error can't find the container with id 8f630cf4b3f50b875bfcfe838267523b3c625451467f39edf2c5d1b5c1f85b38
Sep 09 15:51:48.193960 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:48.193934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr" event={"ID":"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d","Type":"ContainerStarted","Data":"8f630cf4b3f50b875bfcfe838267523b3c625451467f39edf2c5d1b5c1f85b38"}
Sep 09 15:51:48.749465 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:48.749437    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:48.851267 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:48.851234    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"]
Sep 09 15:51:48.851525 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:48.851463    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs" podUID="b776c293-c476-45b8-ac53-ef6aef8cb13f" containerName="maas-api" containerID="cri-o://1b14354122e5f564e503ce6fbb6354d0aed97ef1e12350ed19e422224ef61d44" gracePeriod=30
Sep 09 15:51:48.880945 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:48.880892    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:48.981675 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:48.981642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:49.073659 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.073628    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:49.104970 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.104948    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"
Sep 09 15:51:49.134784 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.134749    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:49.161859 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.161828    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b776c293-c476-45b8-ac53-ef6aef8cb13f-openshift-service-ca\") pod \"b776c293-c476-45b8-ac53-ef6aef8cb13f\" (UID: \"b776c293-c476-45b8-ac53-ef6aef8cb13f\") "
Sep 09 15:51:49.162065 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.161897    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b776c293-c476-45b8-ac53-ef6aef8cb13f-maas-api-tls\") pod \"b776c293-c476-45b8-ac53-ef6aef8cb13f\" (UID: \"b776c293-c476-45b8-ac53-ef6aef8cb13f\") "
Sep 09 15:51:49.162065 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.161960    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b776c293-c476-45b8-ac53-ef6aef8cb13f-metrics-certs\") pod \"b776c293-c476-45b8-ac53-ef6aef8cb13f\" (UID: \"b776c293-c476-45b8-ac53-ef6aef8cb13f\") "
Sep 09 15:51:49.162065 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.162047    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wg5s6\" (UniqueName: \"kubernetes.io/projected/b776c293-c476-45b8-ac53-ef6aef8cb13f-kube-api-access-wg5s6\") pod \"b776c293-c476-45b8-ac53-ef6aef8cb13f\" (UID: \"b776c293-c476-45b8-ac53-ef6aef8cb13f\") "
Sep 09 15:51:49.162240 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.162202    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b776c293-c476-45b8-ac53-ef6aef8cb13f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b776c293-c476-45b8-ac53-ef6aef8cb13f" (UID: "b776c293-c476-45b8-ac53-ef6aef8cb13f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:51:49.162301 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.162270    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b776c293-c476-45b8-ac53-ef6aef8cb13f-openshift-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:51:49.164199 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.164175    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b776c293-c476-45b8-ac53-ef6aef8cb13f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b776c293-c476-45b8-ac53-ef6aef8cb13f" (UID: "b776c293-c476-45b8-ac53-ef6aef8cb13f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:51:49.164288 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.164265    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b776c293-c476-45b8-ac53-ef6aef8cb13f-kube-api-access-wg5s6" (OuterVolumeSpecName: "kube-api-access-wg5s6") pod "b776c293-c476-45b8-ac53-ef6aef8cb13f" (UID: "b776c293-c476-45b8-ac53-ef6aef8cb13f"). InnerVolumeSpecName "kube-api-access-wg5s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:51:49.164337 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.164308    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b776c293-c476-45b8-ac53-ef6aef8cb13f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b776c293-c476-45b8-ac53-ef6aef8cb13f" (UID: "b776c293-c476-45b8-ac53-ef6aef8cb13f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:51:49.201478 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.201418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr" event={"ID":"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d","Type":"ContainerStarted","Data":"4fd120b7cb0b962b1d0bccab43a789c98713cc88b5eff1cb601c0b8afb72cde3"}
Sep 09 15:51:49.202898 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.202868    2578 generic.go:356] "Generic (PLEG): container finished" podID="b776c293-c476-45b8-ac53-ef6aef8cb13f" containerID="1b14354122e5f564e503ce6fbb6354d0aed97ef1e12350ed19e422224ef61d44" exitCode=0
Sep 09 15:51:49.203088 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.202911    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"
Sep 09 15:51:49.203088 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.202934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs" event={"ID":"b776c293-c476-45b8-ac53-ef6aef8cb13f","Type":"ContainerDied","Data":"1b14354122e5f564e503ce6fbb6354d0aed97ef1e12350ed19e422224ef61d44"}
Sep 09 15:51:49.203088 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.202995    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs" event={"ID":"b776c293-c476-45b8-ac53-ef6aef8cb13f","Type":"ContainerDied","Data":"cb06c97179467429ad50ec12b3152009b4ccd5be50aba5df3b944dc8bbc0bfad"}
Sep 09 15:51:49.203088 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.203029    2578 scope.go:117] "RemoveContainer" containerID="1b14354122e5f564e503ce6fbb6354d0aed97ef1e12350ed19e422224ef61d44"
Sep 09 15:51:49.216570 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.216543    2578 scope.go:117] "RemoveContainer" containerID="1b14354122e5f564e503ce6fbb6354d0aed97ef1e12350ed19e422224ef61d44"
Sep 09 15:51:49.216858 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:51:49.216838    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b14354122e5f564e503ce6fbb6354d0aed97ef1e12350ed19e422224ef61d44\": container with ID starting with 1b14354122e5f564e503ce6fbb6354d0aed97ef1e12350ed19e422224ef61d44 not found: ID does not exist" containerID="1b14354122e5f564e503ce6fbb6354d0aed97ef1e12350ed19e422224ef61d44"
Sep 09 15:51:49.216939 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.216871    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b14354122e5f564e503ce6fbb6354d0aed97ef1e12350ed19e422224ef61d44"} err="failed to get container status \"1b14354122e5f564e503ce6fbb6354d0aed97ef1e12350ed19e422224ef61d44\": rpc error: code = NotFound desc = could not find container \"1b14354122e5f564e503ce6fbb6354d0aed97ef1e12350ed19e422224ef61d44\": container with ID starting with 1b14354122e5f564e503ce6fbb6354d0aed97ef1e12350ed19e422224ef61d44 not found: ID does not exist"
Sep 09 15:51:49.222860 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.222802    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr" podStartSLOduration=2.222787787 podStartE2EDuration="2.222787787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:51:49.22006332 +0000 UTC m=+1898.862105309" watchObservedRunningTime="2026-09-09 15:51:49.222787787 +0000 UTC m=+1898.864829775"
Sep 09 15:51:49.235527 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.235499    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"]
Sep 09 15:51:49.238797 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.238771    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ce9689-6fc76456bd-hb5vs"]
Sep 09 15:51:49.263055 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.263013    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg5s6\" (UniqueName: \"kubernetes.io/projected/b776c293-c476-45b8-ac53-ef6aef8cb13f-kube-api-access-wg5s6\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:51:49.263055 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.263036    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b776c293-c476-45b8-ac53-ef6aef8cb13f-maas-api-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:51:49.263055 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.263046    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b776c293-c476-45b8-ac53-ef6aef8cb13f-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:51:49.461831 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:49.461802    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:50.867859 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:50.867817    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b776c293-c476-45b8-ac53-ef6aef8cb13f" path="/var/lib/kubelet/pods/b776c293-c476-45b8-ac53-ef6aef8cb13f/volumes"
Sep 09 15:51:55.088566 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:55.088513    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:55.188708 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:55.188679    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:56.474328 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:56.474293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:56.575623 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:56.575581    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:51:57.868538 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:57.868511    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:57.868996 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:57.868761    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:57.881324 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:57.881297    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:51:58.261576 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:51:58.261549    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:52:06.206529 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:06.206497    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:06.270072 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:06.270036    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:06.371679 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:06.371645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:06.435204 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:06.435175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:06.774083 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:06.774048    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:08.385578 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:08.385533    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:12.455605 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:12.455564    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:12.556052 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:12.555965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:12.731727 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:12.731637    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:15.121209 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:15.121174    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"]
Sep 09 15:52:16.236926 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:16.236893    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"]
Sep 09 15:52:16.439801 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:16.439766    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:16.497687 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:16.497591    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:22.524370 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:22.524307    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:23.481748 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:23.481717    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"]
Sep 09 15:52:23.481965 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:23.481942    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f" podUID="d19c9a9e-2070-4212-9f88-7e0dde8cfd44" containerName="maas-api" containerID="cri-o://bdf97ef0e5e59f7008c035ae695753a28b2b140db5ecb84841ad3603e7732940" gracePeriod=30
Sep 09 15:52:23.507546 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:23.507518    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:23.555119 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:23.555071    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f" podUID="d19c9a9e-2070-4212-9f88-7e0dde8cfd44" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.63:8443/health\": dial tcp 10.132.0.63:8443: connect: connection refused"
Sep 09 15:52:23.736696 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:23.736637    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:52:23.799512 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:23.799485    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-metrics-certs\") pod \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\" (UID: \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\") "
Sep 09 15:52:23.799688 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:23.799550    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9s97\" (UniqueName: \"kubernetes.io/projected/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-kube-api-access-f9s97\") pod \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\" (UID: \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\") "
Sep 09 15:52:23.799688 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:23.799602    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-maas-api-tls\") pod \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\" (UID: \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\") "
Sep 09 15:52:23.799688 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:23.799646    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-openshift-service-ca\") pod \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\" (UID: \"d19c9a9e-2070-4212-9f88-7e0dde8cfd44\") "
Sep 09 15:52:23.800061 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:23.800036    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d19c9a9e-2070-4212-9f88-7e0dde8cfd44" (UID: "d19c9a9e-2070-4212-9f88-7e0dde8cfd44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:52:23.801924 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:23.801898    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-kube-api-access-f9s97" (OuterVolumeSpecName: "kube-api-access-f9s97") pod "d19c9a9e-2070-4212-9f88-7e0dde8cfd44" (UID: "d19c9a9e-2070-4212-9f88-7e0dde8cfd44"). InnerVolumeSpecName "kube-api-access-f9s97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:52:23.801924 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:23.801905    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d19c9a9e-2070-4212-9f88-7e0dde8cfd44" (UID: "d19c9a9e-2070-4212-9f88-7e0dde8cfd44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:52:23.802081 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:23.801925    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d19c9a9e-2070-4212-9f88-7e0dde8cfd44" (UID: "d19c9a9e-2070-4212-9f88-7e0dde8cfd44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:52:23.901447 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:23.901412    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:52:23.901447 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:23.901442    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9s97\" (UniqueName: \"kubernetes.io/projected/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-kube-api-access-f9s97\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:52:23.901447 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:23.901452    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-maas-api-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:52:23.901683 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:23.901461    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19c9a9e-2070-4212-9f88-7e0dde8cfd44-openshift-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:52:24.382280 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:24.382243    2578 generic.go:356] "Generic (PLEG): container finished" podID="d19c9a9e-2070-4212-9f88-7e0dde8cfd44" containerID="bdf97ef0e5e59f7008c035ae695753a28b2b140db5ecb84841ad3603e7732940" exitCode=0
Sep 09 15:52:24.382473 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:24.382292    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"
Sep 09 15:52:24.382473 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:24.382335    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f" event={"ID":"d19c9a9e-2070-4212-9f88-7e0dde8cfd44","Type":"ContainerDied","Data":"bdf97ef0e5e59f7008c035ae695753a28b2b140db5ecb84841ad3603e7732940"}
Sep 09 15:52:24.382473 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:24.382411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f" event={"ID":"d19c9a9e-2070-4212-9f88-7e0dde8cfd44","Type":"ContainerDied","Data":"6d34a00f67d71c27031039e0498f91b2ad2ad368428a1f95d4edab57865806c2"}
Sep 09 15:52:24.382473 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:24.382432    2578 scope.go:117] "RemoveContainer" containerID="bdf97ef0e5e59f7008c035ae695753a28b2b140db5ecb84841ad3603e7732940"
Sep 09 15:52:24.393039 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:24.393017    2578 scope.go:117] "RemoveContainer" containerID="bdf97ef0e5e59f7008c035ae695753a28b2b140db5ecb84841ad3603e7732940"
Sep 09 15:52:24.393340 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:52:24.393323    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf97ef0e5e59f7008c035ae695753a28b2b140db5ecb84841ad3603e7732940\": container with ID starting with bdf97ef0e5e59f7008c035ae695753a28b2b140db5ecb84841ad3603e7732940 not found: ID does not exist" containerID="bdf97ef0e5e59f7008c035ae695753a28b2b140db5ecb84841ad3603e7732940"
Sep 09 15:52:24.393404 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:24.393351    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf97ef0e5e59f7008c035ae695753a28b2b140db5ecb84841ad3603e7732940"} err="failed to get container status \"bdf97ef0e5e59f7008c035ae695753a28b2b140db5ecb84841ad3603e7732940\": rpc error: code = NotFound desc = could not find container \"bdf97ef0e5e59f7008c035ae695753a28b2b140db5ecb84841ad3603e7732940\": container with ID starting with bdf97ef0e5e59f7008c035ae695753a28b2b140db5ecb84841ad3603e7732940 not found: ID does not exist"
Sep 09 15:52:24.407648 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:24.407614    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"]
Sep 09 15:52:24.413856 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:24.413833    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-081ed3-64cfb589c8-jmt9f"]
Sep 09 15:52:24.867183 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:24.867153    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d19c9a9e-2070-4212-9f88-7e0dde8cfd44" path="/var/lib/kubelet/pods/d19c9a9e-2070-4212-9f88-7e0dde8cfd44/volumes"
Sep 09 15:52:26.486222 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:26.486189    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:26.588297 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:26.588261    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:30.144091 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.144048    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr" podUID="65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d" containerName="main" containerID="cri-o://4fd120b7cb0b962b1d0bccab43a789c98713cc88b5eff1cb601c0b8afb72cde3" gracePeriod=45
Sep 09 15:52:30.385390 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.385364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:30.392716 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.392698    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:52:30.417839 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.417756    2578 generic.go:356] "Generic (PLEG): container finished" podID="65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d" containerID="4fd120b7cb0b962b1d0bccab43a789c98713cc88b5eff1cb601c0b8afb72cde3" exitCode=0
Sep 09 15:52:30.417839 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.417796    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"
Sep 09 15:52:30.418053 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.417842    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr" event={"ID":"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d","Type":"ContainerDied","Data":"4fd120b7cb0b962b1d0bccab43a789c98713cc88b5eff1cb601c0b8afb72cde3"}
Sep 09 15:52:30.418053 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.417893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr" event={"ID":"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d","Type":"ContainerDied","Data":"8f630cf4b3f50b875bfcfe838267523b3c625451467f39edf2c5d1b5c1f85b38"}
Sep 09 15:52:30.418053 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.417915    2578 scope.go:117] "RemoveContainer" containerID="4fd120b7cb0b962b1d0bccab43a789c98713cc88b5eff1cb601c0b8afb72cde3"
Sep 09 15:52:30.426675 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.426657    2578 scope.go:117] "RemoveContainer" containerID="4fd120b7cb0b962b1d0bccab43a789c98713cc88b5eff1cb601c0b8afb72cde3"
Sep 09 15:52:30.427014 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:52:30.426989    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd120b7cb0b962b1d0bccab43a789c98713cc88b5eff1cb601c0b8afb72cde3\": container with ID starting with 4fd120b7cb0b962b1d0bccab43a789c98713cc88b5eff1cb601c0b8afb72cde3 not found: ID does not exist" containerID="4fd120b7cb0b962b1d0bccab43a789c98713cc88b5eff1cb601c0b8afb72cde3"
Sep 09 15:52:30.427126 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.427022    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd120b7cb0b962b1d0bccab43a789c98713cc88b5eff1cb601c0b8afb72cde3"} err="failed to get container status \"4fd120b7cb0b962b1d0bccab43a789c98713cc88b5eff1cb601c0b8afb72cde3\": rpc error: code = NotFound desc = could not find container \"4fd120b7cb0b962b1d0bccab43a789c98713cc88b5eff1cb601c0b8afb72cde3\": container with ID starting with 4fd120b7cb0b962b1d0bccab43a789c98713cc88b5eff1cb601c0b8afb72cde3 not found: ID does not exist"
Sep 09 15:52:30.466891 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.466866    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqgbg\" (UniqueName: \"kubernetes.io/projected/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-kube-api-access-cqgbg\") pod \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") "
Sep 09 15:52:30.467074 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.466907    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-home\") pod \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") "
Sep 09 15:52:30.467074 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.466955    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-model-cache\") pod \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") "
Sep 09 15:52:30.467074 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.467008    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-tmp-dir\") pod \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") "
Sep 09 15:52:30.467074 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.467063    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-tls-certs\") pod \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") "
Sep 09 15:52:30.467300 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.467147    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-dshm\") pod \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\" (UID: \"65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d\") "
Sep 09 15:52:30.467300 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.467240    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-home" (OuterVolumeSpecName: "home") pod "65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d" (UID: "65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:52:30.467300 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.467253    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-model-cache" (OuterVolumeSpecName: "model-cache") pod "65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d" (UID: "65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:52:30.467455 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.467329    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d" (UID: "65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:52:30.467683 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.467648    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-home\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:52:30.467952 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.467933    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-model-cache\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:52:30.468283 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.468262    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-tmp-dir\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:52:30.469765 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.469740    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d" (UID: "65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:52:30.469871 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.469856    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-dshm" (OuterVolumeSpecName: "dshm") pod "65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d" (UID: "65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:52:30.469933 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.469892    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-kube-api-access-cqgbg" (OuterVolumeSpecName: "kube-api-access-cqgbg") pod "65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d" (UID: "65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d"). InnerVolumeSpecName "kube-api-access-cqgbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:52:30.569121 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.569089    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-tls-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:52:30.569121 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.569115    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-dshm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:52:30.569121 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.569126    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqgbg\" (UniqueName: \"kubernetes.io/projected/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d-kube-api-access-cqgbg\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:52:30.740664 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.740634    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"]
Sep 09 15:52:30.745426 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.745402    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-b-ef131f3d-kserve-75b67cb466-g4wzr"]
Sep 09 15:52:30.868453 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:30.868423    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d" path="/var/lib/kubelet/pods/65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d/volumes"
Sep 09 15:52:31.258647 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.258603    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf" podUID="343e2e2d-46c8-4f28-a600-663f2e06c467" containerName="main" containerID="cri-o://f608140366ff97532c3c6b6b3fc8c1d5bb5f9375322da5f04fe9332c880c047d" gracePeriod=45
Sep 09 15:52:31.427149 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.427108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf" event={"ID":"343e2e2d-46c8-4f28-a600-663f2e06c467","Type":"ContainerDied","Data":"f608140366ff97532c3c6b6b3fc8c1d5bb5f9375322da5f04fe9332c880c047d"}
Sep 09 15:52:31.427260 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.426896    2578 generic.go:356] "Generic (PLEG): container finished" podID="343e2e2d-46c8-4f28-a600-663f2e06c467" containerID="f608140366ff97532c3c6b6b3fc8c1d5bb5f9375322da5f04fe9332c880c047d" exitCode=0
Sep 09 15:52:31.508764 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.508705    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:52:31.577865 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.577833    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-dshm\") pod \"343e2e2d-46c8-4f28-a600-663f2e06c467\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") "
Sep 09 15:52:31.577865 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.577870    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-tmp-dir\") pod \"343e2e2d-46c8-4f28-a600-663f2e06c467\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") "
Sep 09 15:52:31.578102 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.577898    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-model-cache\") pod \"343e2e2d-46c8-4f28-a600-663f2e06c467\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") "
Sep 09 15:52:31.578102 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.578031    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-home\") pod \"343e2e2d-46c8-4f28-a600-663f2e06c467\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") "
Sep 09 15:52:31.578102 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.578068    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343e2e2d-46c8-4f28-a600-663f2e06c467-tls-certs\") pod \"343e2e2d-46c8-4f28-a600-663f2e06c467\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") "
Sep 09 15:52:31.578221 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.578149    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvlcn\" (UniqueName: \"kubernetes.io/projected/343e2e2d-46c8-4f28-a600-663f2e06c467-kube-api-access-dvlcn\") pod \"343e2e2d-46c8-4f28-a600-663f2e06c467\" (UID: \"343e2e2d-46c8-4f28-a600-663f2e06c467\") "
Sep 09 15:52:31.578221 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.578182    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-model-cache" (OuterVolumeSpecName: "model-cache") pod "343e2e2d-46c8-4f28-a600-663f2e06c467" (UID: "343e2e2d-46c8-4f28-a600-663f2e06c467"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:52:31.578221 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.578195    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "343e2e2d-46c8-4f28-a600-663f2e06c467" (UID: "343e2e2d-46c8-4f28-a600-663f2e06c467"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:52:31.578390 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.578278    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-home" (OuterVolumeSpecName: "home") pod "343e2e2d-46c8-4f28-a600-663f2e06c467" (UID: "343e2e2d-46c8-4f28-a600-663f2e06c467"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:52:31.578448 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.578422    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-home\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:52:31.578448 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.578438    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-tmp-dir\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:52:31.578556 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.578450    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-model-cache\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:52:31.580226 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.580205    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-dshm" (OuterVolumeSpecName: "dshm") pod "343e2e2d-46c8-4f28-a600-663f2e06c467" (UID: "343e2e2d-46c8-4f28-a600-663f2e06c467"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:52:31.580310 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.580271    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/343e2e2d-46c8-4f28-a600-663f2e06c467-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "343e2e2d-46c8-4f28-a600-663f2e06c467" (UID: "343e2e2d-46c8-4f28-a600-663f2e06c467"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:52:31.580403 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.580385    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343e2e2d-46c8-4f28-a600-663f2e06c467-kube-api-access-dvlcn" (OuterVolumeSpecName: "kube-api-access-dvlcn") pod "343e2e2d-46c8-4f28-a600-663f2e06c467" (UID: "343e2e2d-46c8-4f28-a600-663f2e06c467"). InnerVolumeSpecName "kube-api-access-dvlcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:52:31.679797 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.679766    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343e2e2d-46c8-4f28-a600-663f2e06c467-dshm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:52:31.679797 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.679792    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343e2e2d-46c8-4f28-a600-663f2e06c467-tls-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:52:31.679797 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.679803    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvlcn\" (UniqueName: \"kubernetes.io/projected/343e2e2d-46c8-4f28-a600-663f2e06c467-kube-api-access-dvlcn\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:52:31.755686 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.755657    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:31.999612 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:31.999583    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:52:32.433951 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:32.433924    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"
Sep 09 15:52:32.433951 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:32.433947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf" event={"ID":"343e2e2d-46c8-4f28-a600-663f2e06c467","Type":"ContainerDied","Data":"0f4cc886076d1ded85c95713c8d6adc185b19e4ad5a860c275e3eaedce1c42bf"}
Sep 09 15:52:32.434474 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:32.434024    2578 scope.go:117] "RemoveContainer" containerID="f608140366ff97532c3c6b6b3fc8c1d5bb5f9375322da5f04fe9332c880c047d"
Sep 09 15:52:32.455073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:32.455041    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"]
Sep 09 15:52:32.458277 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:32.458252    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/e2e-conflict-a-ef131f3d-kserve-844d686668-2z2jf"]
Sep 09 15:52:32.867833 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:32.867755    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="343e2e2d-46c8-4f28-a600-663f2e06c467" path="/var/lib/kubelet/pods/343e2e2d-46c8-4f28-a600-663f2e06c467/volumes"
Sep 09 15:52:37.267761 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:52:37.267702    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:53:17.893836 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:17.893802    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:53:23.195535 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:23.195496    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:53:23.250026 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:23.249950    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:53:43.667070 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.667033    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7463-80a7fb"]
Sep 09 15:53:43.667542 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.667437    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="343e2e2d-46c8-4f28-a600-663f2e06c467" containerName="main"
Sep 09 15:53:43.667542 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.667448    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="343e2e2d-46c8-4f28-a600-663f2e06c467" containerName="main"
Sep 09 15:53:43.667542 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.667465    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d19c9a9e-2070-4212-9f88-7e0dde8cfd44" containerName="maas-api"
Sep 09 15:53:43.667542 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.667470    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d19c9a9e-2070-4212-9f88-7e0dde8cfd44" containerName="maas-api"
Sep 09 15:53:43.667542 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.667477    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d" containerName="main"
Sep 09 15:53:43.667542 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.667482    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d" containerName="main"
Sep 09 15:53:43.667542 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.667500    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b776c293-c476-45b8-ac53-ef6aef8cb13f" containerName="maas-api"
Sep 09 15:53:43.667542 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.667505    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b776c293-c476-45b8-ac53-ef6aef8cb13f" containerName="maas-api"
Sep 09 15:53:43.667796 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.667561    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b776c293-c476-45b8-ac53-ef6aef8cb13f" containerName="maas-api"
Sep 09 15:53:43.667796 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.667569    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="65a4a882-9e17-4dfb-8a3d-45eabcf6bd7d" containerName="main"
Sep 09 15:53:43.667796 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.667577    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="343e2e2d-46c8-4f28-a600-663f2e06c467" containerName="main"
Sep 09 15:53:43.667796 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.667583    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d19c9a9e-2070-4212-9f88-7e0dde8cfd44" containerName="maas-api"
Sep 09 15:53:43.670725 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.670710    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7463-80a7fb"
Sep 09 15:53:43.673203 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.673188    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pcz7f\""
Sep 09 15:53:43.676514 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.676490    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7463-80a7fb"]
Sep 09 15:53:43.768143 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.768110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sstdb\" (UniqueName: \"kubernetes.io/projected/815b724c-f1b2-4ff3-bda9-cfbfb22304be-kube-api-access-sstdb\") pod \"mt-curl-7463-80a7fb\" (UID: \"815b724c-f1b2-4ff3-bda9-cfbfb22304be\") " pod="opendatahub/mt-curl-7463-80a7fb"
Sep 09 15:53:43.869523 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.869493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sstdb\" (UniqueName: \"kubernetes.io/projected/815b724c-f1b2-4ff3-bda9-cfbfb22304be-kube-api-access-sstdb\") pod \"mt-curl-7463-80a7fb\" (UID: \"815b724c-f1b2-4ff3-bda9-cfbfb22304be\") " pod="opendatahub/mt-curl-7463-80a7fb"
Sep 09 15:53:43.877928 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.877901    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sstdb\" (UniqueName: \"kubernetes.io/projected/815b724c-f1b2-4ff3-bda9-cfbfb22304be-kube-api-access-sstdb\") pod \"mt-curl-7463-80a7fb\" (UID: \"815b724c-f1b2-4ff3-bda9-cfbfb22304be\") " pod="opendatahub/mt-curl-7463-80a7fb"
Sep 09 15:53:43.981336 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:43.981245    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7463-80a7fb"
Sep 09 15:53:44.251329 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:44.251279    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7463-80a7fb"]
Sep 09 15:53:44.253715 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:53:44.253685    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod815b724c_f1b2_4ff3_bda9_cfbfb22304be.slice/crio-e818276b7bd18529d1c7a752323c89e95f00422c06a2bc22c0ce7c7e89b51ef0 WatchSource:0}: Error finding container e818276b7bd18529d1c7a752323c89e95f00422c06a2bc22c0ce7c7e89b51ef0: Status 404 returned error can't find the container with id e818276b7bd18529d1c7a752323c89e95f00422c06a2bc22c0ce7c7e89b51ef0
Sep 09 15:53:44.801364 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:44.801329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7463-80a7fb" event={"ID":"815b724c-f1b2-4ff3-bda9-cfbfb22304be","Type":"ContainerStarted","Data":"e818276b7bd18529d1c7a752323c89e95f00422c06a2bc22c0ce7c7e89b51ef0"}
Sep 09 15:53:45.809211 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:45.809180    2578 generic.go:356] "Generic (PLEG): container finished" podID="815b724c-f1b2-4ff3-bda9-cfbfb22304be" containerID="6eac61ebe20ca698a00e411fbb070dad238a38e6df8bff87d6318e6528405da5" exitCode=0
Sep 09 15:53:45.809606 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:45.809265    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7463-80a7fb" event={"ID":"815b724c-f1b2-4ff3-bda9-cfbfb22304be","Type":"ContainerDied","Data":"6eac61ebe20ca698a00e411fbb070dad238a38e6df8bff87d6318e6528405da5"}
Sep 09 15:53:46.946037 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:46.946005    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7463-80a7fb"
Sep 09 15:53:46.965605 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:46.965584    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7463-80a7fb_815b724c-f1b2-4ff3-bda9-cfbfb22304be/mt-curl-7463-80a7fb/0.log"
Sep 09 15:53:46.987789 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:46.987762    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7463-80a7fb"]
Sep 09 15:53:46.991837 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:46.991815    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7463-80a7fb"]
Sep 09 15:53:47.003178 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:47.003157    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sstdb\" (UniqueName: \"kubernetes.io/projected/815b724c-f1b2-4ff3-bda9-cfbfb22304be-kube-api-access-sstdb\") pod \"815b724c-f1b2-4ff3-bda9-cfbfb22304be\" (UID: \"815b724c-f1b2-4ff3-bda9-cfbfb22304be\") "
Sep 09 15:53:47.005322 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:47.005291    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815b724c-f1b2-4ff3-bda9-cfbfb22304be-kube-api-access-sstdb" (OuterVolumeSpecName: "kube-api-access-sstdb") pod "815b724c-f1b2-4ff3-bda9-cfbfb22304be" (UID: "815b724c-f1b2-4ff3-bda9-cfbfb22304be"). InnerVolumeSpecName "kube-api-access-sstdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:53:47.104072 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:47.103962    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sstdb\" (UniqueName: \"kubernetes.io/projected/815b724c-f1b2-4ff3-bda9-cfbfb22304be-kube-api-access-sstdb\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:53:47.819740 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:47.819708    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7463-80a7fb"
Sep 09 15:53:47.819919 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:47.819755    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e818276b7bd18529d1c7a752323c89e95f00422c06a2bc22c0ce7c7e89b51ef0"
Sep 09 15:53:47.829607 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:53:47.829581    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7463-80a7fb\" is forbidden: User \"system:node:ip-10-0-134-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-183.ec2.internal' and this object" podUID="815b724c-f1b2-4ff3-bda9-cfbfb22304be" pod="opendatahub/mt-curl-7463-80a7fb"
Sep 09 15:53:48.454565 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:48.454532    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:53:48.867541 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:48.867465    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="815b724c-f1b2-4ff3-bda9-cfbfb22304be" path="/var/lib/kubelet/pods/815b724c-f1b2-4ff3-bda9-cfbfb22304be/volumes"
Sep 09 15:53:50.055128 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:50.055097    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:53:58.487399 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:53:58.487369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:54:03.480223 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:03.480184    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"]
Sep 09 15:54:03.488574 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:03.488544    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"]
Sep 09 15:54:03.615912 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:03.615882    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:54:11.858727 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:11.858696    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"]
Sep 09 15:54:11.859229 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:11.859212    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="815b724c-f1b2-4ff3-bda9-cfbfb22304be" containerName="mt-curl-7463-80a7fb"
Sep 09 15:54:11.859280 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:11.859232    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="815b724c-f1b2-4ff3-bda9-cfbfb22304be" containerName="mt-curl-7463-80a7fb"
Sep 09 15:54:11.859316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:11.859302    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="815b724c-f1b2-4ff3-bda9-cfbfb22304be" containerName="mt-curl-7463-80a7fb"
Sep 09 15:54:11.862265 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:11.862249    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:11.865746 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:11.865725    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-450a51\"/\"default-dockercfg-lqbr4\""
Sep 09 15:54:11.865877 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:11.865725    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-450a51\"/\"openshift-service-ca.crt\""
Sep 09 15:54:11.865877 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:11.865725    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-450a51\"/\"sub-test-model-450a51-kserve-self-signed-certs\""
Sep 09 15:54:11.865877 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:11.865773    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-450a51\"/\"kube-root-ca.crt\""
Sep 09 15:54:11.870803 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:11.870780    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"]
Sep 09 15:54:11.884390 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:11.884352    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:54:12.057681 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.057642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-tmp-dir\") pod \"sub-test-model-450a51-kserve-f9d5dc755-vkd57\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") " pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.057885 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.057692    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee622e7-fe25-413b-b475-550ca9bcd3d7-tls-certs\") pod \"sub-test-model-450a51-kserve-f9d5dc755-vkd57\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") " pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.057885 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.057727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-model-cache\") pod \"sub-test-model-450a51-kserve-f9d5dc755-vkd57\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") " pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.057885 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.057784    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-dshm\") pod \"sub-test-model-450a51-kserve-f9d5dc755-vkd57\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") " pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.057885 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.057821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfd5q\" (UniqueName: \"kubernetes.io/projected/7ee622e7-fe25-413b-b475-550ca9bcd3d7-kube-api-access-tfd5q\") pod \"sub-test-model-450a51-kserve-f9d5dc755-vkd57\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") " pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.058124 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.057919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-home\") pod \"sub-test-model-450a51-kserve-f9d5dc755-vkd57\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") " pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.158714 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.158677    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-model-cache\") pod \"sub-test-model-450a51-kserve-f9d5dc755-vkd57\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") " pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.158896 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.158731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-dshm\") pod \"sub-test-model-450a51-kserve-f9d5dc755-vkd57\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") " pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.158896 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.158761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfd5q\" (UniqueName: \"kubernetes.io/projected/7ee622e7-fe25-413b-b475-550ca9bcd3d7-kube-api-access-tfd5q\") pod \"sub-test-model-450a51-kserve-f9d5dc755-vkd57\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") " pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.158896 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.158785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-home\") pod \"sub-test-model-450a51-kserve-f9d5dc755-vkd57\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") " pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.159101 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.158898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-tmp-dir\") pod \"sub-test-model-450a51-kserve-f9d5dc755-vkd57\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") " pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.159101 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.158926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee622e7-fe25-413b-b475-550ca9bcd3d7-tls-certs\") pod \"sub-test-model-450a51-kserve-f9d5dc755-vkd57\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") " pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.159218 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.159116    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-model-cache\") pod \"sub-test-model-450a51-kserve-f9d5dc755-vkd57\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") " pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.159272 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.159224    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-home\") pod \"sub-test-model-450a51-kserve-f9d5dc755-vkd57\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") " pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.159272 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.159251    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-tmp-dir\") pod \"sub-test-model-450a51-kserve-f9d5dc755-vkd57\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") " pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.161246 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.161224    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-dshm\") pod \"sub-test-model-450a51-kserve-f9d5dc755-vkd57\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") " pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.161457 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.161439    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee622e7-fe25-413b-b475-550ca9bcd3d7-tls-certs\") pod \"sub-test-model-450a51-kserve-f9d5dc755-vkd57\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") " pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.167172 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.167153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfd5q\" (UniqueName: \"kubernetes.io/projected/7ee622e7-fe25-413b-b475-550ca9bcd3d7-kube-api-access-tfd5q\") pod \"sub-test-model-450a51-kserve-f9d5dc755-vkd57\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") " pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.172041 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.172023    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:12.442953 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.442927    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"]
Sep 09 15:54:12.444942 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:54:12.444900    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ee622e7_fe25_413b_b475_550ca9bcd3d7.slice/crio-2e4d692bf3f7ae6aba41115e1485be252b66aa470873ecf42f2afb689668ce25 WatchSource:0}: Error finding container 2e4d692bf3f7ae6aba41115e1485be252b66aa470873ecf42f2afb689668ce25: Status 404 returned error can't find the container with id 2e4d692bf3f7ae6aba41115e1485be252b66aa470873ecf42f2afb689668ce25
Sep 09 15:54:12.954565 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.954533    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57" event={"ID":"7ee622e7-fe25-413b-b475-550ca9bcd3d7","Type":"ContainerStarted","Data":"34204900f35bcb2936c3b539d4e9b633886afa2b1a4c63a840ec84df89aca058"}
Sep 09 15:54:12.954565 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.954572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57" event={"ID":"7ee622e7-fe25-413b-b475-550ca9bcd3d7","Type":"ContainerStarted","Data":"2e4d692bf3f7ae6aba41115e1485be252b66aa470873ecf42f2afb689668ce25"}
Sep 09 15:54:12.971622 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:12.971573    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57" podStartSLOduration=1.971559263 podStartE2EDuration="1.971559263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:54:12.969818257 +0000 UTC m=+2042.611860257" watchObservedRunningTime="2026-09-09 15:54:12.971559263 +0000 UTC m=+2042.613601259"
Sep 09 15:54:14.257659 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.257616    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"]
Sep 09 15:54:14.258117 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.257829    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq" podUID="03f63f69-2753-4d7d-9fd5-43534cc5db99" containerName="maas-api" containerID="cri-o://8a090b1646ad95220ce3dda3b0eb61406effc0682b8d0ef544fb20452a9b1832" gracePeriod=30
Sep 09 15:54:14.276332 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.276301    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:54:14.348283 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.348249    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:54:14.410563 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.410534    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:54:14.490992 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.490947    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:54:14.518933 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.518871    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:54:14.548767 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.548732    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:54:14.582013 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.581952    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f63f69-2753-4d7d-9fd5-43534cc5db99-openshift-service-ca\") pod \"03f63f69-2753-4d7d-9fd5-43534cc5db99\" (UID: \"03f63f69-2753-4d7d-9fd5-43534cc5db99\") "
Sep 09 15:54:14.582196 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.582048    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f63f69-2753-4d7d-9fd5-43534cc5db99-metrics-certs\") pod \"03f63f69-2753-4d7d-9fd5-43534cc5db99\" (UID: \"03f63f69-2753-4d7d-9fd5-43534cc5db99\") "
Sep 09 15:54:14.582196 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.582109    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f63f69-2753-4d7d-9fd5-43534cc5db99-maas-api-tls\") pod \"03f63f69-2753-4d7d-9fd5-43534cc5db99\" (UID: \"03f63f69-2753-4d7d-9fd5-43534cc5db99\") "
Sep 09 15:54:14.582196 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.582173    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fncrh\" (UniqueName: \"kubernetes.io/projected/03f63f69-2753-4d7d-9fd5-43534cc5db99-kube-api-access-fncrh\") pod \"03f63f69-2753-4d7d-9fd5-43534cc5db99\" (UID: \"03f63f69-2753-4d7d-9fd5-43534cc5db99\") "
Sep 09 15:54:14.582393 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.582368    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f63f69-2753-4d7d-9fd5-43534cc5db99-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "03f63f69-2753-4d7d-9fd5-43534cc5db99" (UID: "03f63f69-2753-4d7d-9fd5-43534cc5db99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:54:14.582523 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.582507    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f63f69-2753-4d7d-9fd5-43534cc5db99-openshift-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:54:14.584408 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.584383    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f63f69-2753-4d7d-9fd5-43534cc5db99-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "03f63f69-2753-4d7d-9fd5-43534cc5db99" (UID: "03f63f69-2753-4d7d-9fd5-43534cc5db99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:54:14.584408 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.584403    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f63f69-2753-4d7d-9fd5-43534cc5db99-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "03f63f69-2753-4d7d-9fd5-43534cc5db99" (UID: "03f63f69-2753-4d7d-9fd5-43534cc5db99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:54:14.584576 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.584407    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f63f69-2753-4d7d-9fd5-43534cc5db99-kube-api-access-fncrh" (OuterVolumeSpecName: "kube-api-access-fncrh") pod "03f63f69-2753-4d7d-9fd5-43534cc5db99" (UID: "03f63f69-2753-4d7d-9fd5-43534cc5db99"). InnerVolumeSpecName "kube-api-access-fncrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:54:14.683257 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.683226    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f63f69-2753-4d7d-9fd5-43534cc5db99-maas-api-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:54:14.683257 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.683259    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fncrh\" (UniqueName: \"kubernetes.io/projected/03f63f69-2753-4d7d-9fd5-43534cc5db99-kube-api-access-fncrh\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:54:14.683454 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.683271    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f63f69-2753-4d7d-9fd5-43534cc5db99-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:54:14.952019 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.951990    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:54:14.965892 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.965848    2578 generic.go:356] "Generic (PLEG): container finished" podID="03f63f69-2753-4d7d-9fd5-43534cc5db99" containerID="8a090b1646ad95220ce3dda3b0eb61406effc0682b8d0ef544fb20452a9b1832" exitCode=0
Sep 09 15:54:14.966072 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.965902    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"
Sep 09 15:54:14.966072 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.965930    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq" event={"ID":"03f63f69-2753-4d7d-9fd5-43534cc5db99","Type":"ContainerDied","Data":"8a090b1646ad95220ce3dda3b0eb61406effc0682b8d0ef544fb20452a9b1832"}
Sep 09 15:54:14.966072 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.966022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq" event={"ID":"03f63f69-2753-4d7d-9fd5-43534cc5db99","Type":"ContainerDied","Data":"a6e3e103baadabfab0ca8fc72df461ffc051bfc6c62525f0a1c2762540185773"}
Sep 09 15:54:14.966072 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.966045    2578 scope.go:117] "RemoveContainer" containerID="8a090b1646ad95220ce3dda3b0eb61406effc0682b8d0ef544fb20452a9b1832"
Sep 09 15:54:14.975828 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.975813    2578 scope.go:117] "RemoveContainer" containerID="8a090b1646ad95220ce3dda3b0eb61406effc0682b8d0ef544fb20452a9b1832"
Sep 09 15:54:14.976111 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:54:14.976094    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a090b1646ad95220ce3dda3b0eb61406effc0682b8d0ef544fb20452a9b1832\": container with ID starting with 8a090b1646ad95220ce3dda3b0eb61406effc0682b8d0ef544fb20452a9b1832 not found: ID does not exist" containerID="8a090b1646ad95220ce3dda3b0eb61406effc0682b8d0ef544fb20452a9b1832"
Sep 09 15:54:14.976161 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.976118    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a090b1646ad95220ce3dda3b0eb61406effc0682b8d0ef544fb20452a9b1832"} err="failed to get container status \"8a090b1646ad95220ce3dda3b0eb61406effc0682b8d0ef544fb20452a9b1832\": rpc error: code = NotFound desc = could not find container \"8a090b1646ad95220ce3dda3b0eb61406effc0682b8d0ef544fb20452a9b1832\": container with ID starting with 8a090b1646ad95220ce3dda3b0eb61406effc0682b8d0ef544fb20452a9b1832 not found: ID does not exist"
Sep 09 15:54:14.983142 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.983096    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"]
Sep 09 15:54:14.985287 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:14.985267    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1e15c8-5999cf6cb4-qs8tq"]
Sep 09 15:54:16.867651 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:16.867622    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03f63f69-2753-4d7d-9fd5-43534cc5db99" path="/var/lib/kubelet/pods/03f63f69-2753-4d7d-9fd5-43534cc5db99/volumes"
Sep 09 15:54:18.500932 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.500889    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd" podUID="f5f94d58-8121-49d0-96ed-4890dd50b00b" containerName="main" containerID="cri-o://82766adac26ecd276fd30cbb3a8af0f831b08ed48f6a13a5ee744ed0791d6e59" gracePeriod=45
Sep 09 15:54:18.508700 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.508654    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf" podUID="10a7a5ef-7ff3-42be-862f-8dd4527eecd8" containerName="main" containerID="cri-o://256b6992d791512a312721fbd754ff524299b695d6f1821f778a266cd7e3ff7c" gracePeriod=45
Sep 09 15:54:18.752886 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.752826    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:54:18.778812 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.778792    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:54:18.823030 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823003    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt2hb\" (UniqueName: \"kubernetes.io/projected/f5f94d58-8121-49d0-96ed-4890dd50b00b-kube-api-access-xt2hb\") pod \"f5f94d58-8121-49d0-96ed-4890dd50b00b\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") "
Sep 09 15:54:18.823217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823044    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-home\") pod \"f5f94d58-8121-49d0-96ed-4890dd50b00b\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") "
Sep 09 15:54:18.823217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823069    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-dshm\") pod \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") "
Sep 09 15:54:18.823217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823087    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-dshm\") pod \"f5f94d58-8121-49d0-96ed-4890dd50b00b\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") "
Sep 09 15:54:18.823217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823155    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-tmp-dir\") pod \"f5f94d58-8121-49d0-96ed-4890dd50b00b\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") "
Sep 09 15:54:18.823217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823183    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5nvl\" (UniqueName: \"kubernetes.io/projected/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-kube-api-access-v5nvl\") pod \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") "
Sep 09 15:54:18.823217 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823216    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-model-cache\") pod \"f5f94d58-8121-49d0-96ed-4890dd50b00b\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") "
Sep 09 15:54:18.823547 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823256    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f94d58-8121-49d0-96ed-4890dd50b00b-tls-certs\") pod \"f5f94d58-8121-49d0-96ed-4890dd50b00b\" (UID: \"f5f94d58-8121-49d0-96ed-4890dd50b00b\") "
Sep 09 15:54:18.823547 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823284    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-model-cache\") pod \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") "
Sep 09 15:54:18.823547 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823316    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-home" (OuterVolumeSpecName: "home") pod "f5f94d58-8121-49d0-96ed-4890dd50b00b" (UID: "f5f94d58-8121-49d0-96ed-4890dd50b00b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.823547 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823338    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-tmp-dir\") pod \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") "
Sep 09 15:54:18.823547 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823373    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-home\") pod \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") "
Sep 09 15:54:18.823547 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823423    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-tls-certs\") pod \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\" (UID: \"10a7a5ef-7ff3-42be-862f-8dd4527eecd8\") "
Sep 09 15:54:18.823547 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823522    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5f94d58-8121-49d0-96ed-4890dd50b00b" (UID: "f5f94d58-8121-49d0-96ed-4890dd50b00b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.823894 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823622    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-model-cache" (OuterVolumeSpecName: "model-cache") pod "10a7a5ef-7ff3-42be-862f-8dd4527eecd8" (UID: "10a7a5ef-7ff3-42be-862f-8dd4527eecd8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.823894 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823660    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5f94d58-8121-49d0-96ed-4890dd50b00b" (UID: "f5f94d58-8121-49d0-96ed-4890dd50b00b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.823894 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823859    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-tmp-dir\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:54:18.823894 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823882    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-model-cache\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:54:18.823894 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823895    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-model-cache\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:54:18.824202 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823907    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-home\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:54:18.824202 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.823943    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10a7a5ef-7ff3-42be-862f-8dd4527eecd8" (UID: "10a7a5ef-7ff3-42be-862f-8dd4527eecd8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.824202 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.824113    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-home" (OuterVolumeSpecName: "home") pod "10a7a5ef-7ff3-42be-862f-8dd4527eecd8" (UID: "10a7a5ef-7ff3-42be-862f-8dd4527eecd8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.825939 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.825909    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-dshm" (OuterVolumeSpecName: "dshm") pod "f5f94d58-8121-49d0-96ed-4890dd50b00b" (UID: "f5f94d58-8121-49d0-96ed-4890dd50b00b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.826095 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.826011    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f94d58-8121-49d0-96ed-4890dd50b00b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5f94d58-8121-49d0-96ed-4890dd50b00b" (UID: "f5f94d58-8121-49d0-96ed-4890dd50b00b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:54:18.826427 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.826402    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10a7a5ef-7ff3-42be-862f-8dd4527eecd8" (UID: "10a7a5ef-7ff3-42be-862f-8dd4527eecd8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:54:18.826548 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.826517    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-kube-api-access-v5nvl" (OuterVolumeSpecName: "kube-api-access-v5nvl") pod "10a7a5ef-7ff3-42be-862f-8dd4527eecd8" (UID: "10a7a5ef-7ff3-42be-862f-8dd4527eecd8"). InnerVolumeSpecName "kube-api-access-v5nvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:54:18.826754 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.826729    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-dshm" (OuterVolumeSpecName: "dshm") pod "10a7a5ef-7ff3-42be-862f-8dd4527eecd8" (UID: "10a7a5ef-7ff3-42be-862f-8dd4527eecd8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:54:18.826830 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.826789    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f94d58-8121-49d0-96ed-4890dd50b00b-kube-api-access-xt2hb" (OuterVolumeSpecName: "kube-api-access-xt2hb") pod "f5f94d58-8121-49d0-96ed-4890dd50b00b" (UID: "f5f94d58-8121-49d0-96ed-4890dd50b00b"). InnerVolumeSpecName "kube-api-access-xt2hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:54:18.924788 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.924748    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-dshm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:54:18.924788 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.924780    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f94d58-8121-49d0-96ed-4890dd50b00b-dshm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:54:18.925009 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.924795    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5nvl\" (UniqueName: \"kubernetes.io/projected/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-kube-api-access-v5nvl\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:54:18.925009 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.924808    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f94d58-8121-49d0-96ed-4890dd50b00b-tls-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:54:18.925009 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.924822    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-tmp-dir\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:54:18.925009 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.924833    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-home\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:54:18.925009 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.924845    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a7a5ef-7ff3-42be-862f-8dd4527eecd8-tls-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:54:18.925009 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.924858    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt2hb\" (UniqueName: \"kubernetes.io/projected/f5f94d58-8121-49d0-96ed-4890dd50b00b-kube-api-access-xt2hb\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:54:18.991901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.991868    2578 generic.go:356] "Generic (PLEG): container finished" podID="10a7a5ef-7ff3-42be-862f-8dd4527eecd8" containerID="256b6992d791512a312721fbd754ff524299b695d6f1821f778a266cd7e3ff7c" exitCode=0
Sep 09 15:54:18.992088 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.991909    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"
Sep 09 15:54:18.992088 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.991947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf" event={"ID":"10a7a5ef-7ff3-42be-862f-8dd4527eecd8","Type":"ContainerDied","Data":"256b6992d791512a312721fbd754ff524299b695d6f1821f778a266cd7e3ff7c"}
Sep 09 15:54:18.992088 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.992004    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf" event={"ID":"10a7a5ef-7ff3-42be-862f-8dd4527eecd8","Type":"ContainerDied","Data":"923d6e54b4bc0d25497bd5d09a3cc3d74c6d3dea8071021e0dfd3313cae4e9c9"}
Sep 09 15:54:18.992088 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.992022    2578 scope.go:117] "RemoveContainer" containerID="256b6992d791512a312721fbd754ff524299b695d6f1821f778a266cd7e3ff7c"
Sep 09 15:54:18.993477 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.993444    2578 generic.go:356] "Generic (PLEG): container finished" podID="f5f94d58-8121-49d0-96ed-4890dd50b00b" containerID="82766adac26ecd276fd30cbb3a8af0f831b08ed48f6a13a5ee744ed0791d6e59" exitCode=0
Sep 09 15:54:18.993590 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.993480    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"
Sep 09 15:54:18.993590 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.993491    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd" event={"ID":"f5f94d58-8121-49d0-96ed-4890dd50b00b","Type":"ContainerDied","Data":"82766adac26ecd276fd30cbb3a8af0f831b08ed48f6a13a5ee744ed0791d6e59"}
Sep 09 15:54:18.993590 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:18.993520    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd" event={"ID":"f5f94d58-8121-49d0-96ed-4890dd50b00b","Type":"ContainerDied","Data":"2af50c2e66a3523ce476a9a0f36f5a5dd108053fefa8268c347159ba64ea29cc"}
Sep 09 15:54:19.000886 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:19.000866    2578 scope.go:117] "RemoveContainer" containerID="256b6992d791512a312721fbd754ff524299b695d6f1821f778a266cd7e3ff7c"
Sep 09 15:54:19.001167 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:54:19.001148    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"256b6992d791512a312721fbd754ff524299b695d6f1821f778a266cd7e3ff7c\": container with ID starting with 256b6992d791512a312721fbd754ff524299b695d6f1821f778a266cd7e3ff7c not found: ID does not exist" containerID="256b6992d791512a312721fbd754ff524299b695d6f1821f778a266cd7e3ff7c"
Sep 09 15:54:19.001254 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:19.001180    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"256b6992d791512a312721fbd754ff524299b695d6f1821f778a266cd7e3ff7c"} err="failed to get container status \"256b6992d791512a312721fbd754ff524299b695d6f1821f778a266cd7e3ff7c\": rpc error: code = NotFound desc = could not find container \"256b6992d791512a312721fbd754ff524299b695d6f1821f778a266cd7e3ff7c\": container with ID starting with 256b6992d791512a312721fbd754ff524299b695d6f1821f778a266cd7e3ff7c not found: ID does not exist"
Sep 09 15:54:19.001254 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:19.001203    2578 scope.go:117] "RemoveContainer" containerID="82766adac26ecd276fd30cbb3a8af0f831b08ed48f6a13a5ee744ed0791d6e59"
Sep 09 15:54:19.009389 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:19.009339    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"]
Sep 09 15:54:19.010387 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:19.010364    2578 scope.go:117] "RemoveContainer" containerID="82766adac26ecd276fd30cbb3a8af0f831b08ed48f6a13a5ee744ed0791d6e59"
Sep 09 15:54:19.010660 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:54:19.010640    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82766adac26ecd276fd30cbb3a8af0f831b08ed48f6a13a5ee744ed0791d6e59\": container with ID starting with 82766adac26ecd276fd30cbb3a8af0f831b08ed48f6a13a5ee744ed0791d6e59 not found: ID does not exist" containerID="82766adac26ecd276fd30cbb3a8af0f831b08ed48f6a13a5ee744ed0791d6e59"
Sep 09 15:54:19.010727 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:19.010667    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82766adac26ecd276fd30cbb3a8af0f831b08ed48f6a13a5ee744ed0791d6e59"} err="failed to get container status \"82766adac26ecd276fd30cbb3a8af0f831b08ed48f6a13a5ee744ed0791d6e59\": rpc error: code = NotFound desc = could not find container \"82766adac26ecd276fd30cbb3a8af0f831b08ed48f6a13a5ee744ed0791d6e59\": container with ID starting with 82766adac26ecd276fd30cbb3a8af0f831b08ed48f6a13a5ee744ed0791d6e59 not found: ID does not exist"
Sep 09 15:54:19.013940 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:19.013922    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/embedding-w4-1e15c8-kserve-656dbcb9bf-v4pzd"]
Sep 09 15:54:19.024472 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:19.024452    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"]
Sep 09 15:54:19.028693 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:19.028675    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1e15c8/premium-w4-1e15c8-kserve-8779cb4d9-hdtnf"]
Sep 09 15:54:20.737857 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:20.737825    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:54:20.867094 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:20.867051    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10a7a5ef-7ff3-42be-862f-8dd4527eecd8" path="/var/lib/kubelet/pods/10a7a5ef-7ff3-42be-862f-8dd4527eecd8/volumes"
Sep 09 15:54:20.867572 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:20.867551    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5f94d58-8121-49d0-96ed-4890dd50b00b" path="/var/lib/kubelet/pods/f5f94d58-8121-49d0-96ed-4890dd50b00b/volumes"
Sep 09 15:54:22.172747 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:22.172709    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:22.173180 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:22.172961    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:22.185034 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:22.185017    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:23.040076 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:23.040051    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:54:23.463020 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:23.462973    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:54:23.565924 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:23.565895    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:54:35.937921 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:35.937843    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:54:40.632419 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:40.632390    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:54:40.746362 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:40.746335    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:54:41.405127 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:41.405097    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:54:50.488089 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:50.488053    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:54:55.702203 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:54:55.702170    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:01.093000 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:01.092943    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:05.971240 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:05.971200    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:15.247929 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:15.247901    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 15:55:15.253122 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:15.253097    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 15:55:15.255021 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:15.255000    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:55:15.260292 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:15.260276    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 15:55:23.983592 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:23.983561    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7463-cc5045"]
Sep 09 15:55:23.984196 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:23.983940    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f63f69-2753-4d7d-9fd5-43534cc5db99" containerName="maas-api"
Sep 09 15:55:23.984196 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:23.983950    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f63f69-2753-4d7d-9fd5-43534cc5db99" containerName="maas-api"
Sep 09 15:55:23.984196 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:23.983966    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a7a5ef-7ff3-42be-862f-8dd4527eecd8" containerName="main"
Sep 09 15:55:23.984196 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:23.983971    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a7a5ef-7ff3-42be-862f-8dd4527eecd8" containerName="main"
Sep 09 15:55:23.984196 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:23.983999    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f94d58-8121-49d0-96ed-4890dd50b00b" containerName="main"
Sep 09 15:55:23.984196 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:23.984007    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f94d58-8121-49d0-96ed-4890dd50b00b" containerName="main"
Sep 09 15:55:23.984196 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:23.984077    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="03f63f69-2753-4d7d-9fd5-43534cc5db99" containerName="maas-api"
Sep 09 15:55:23.984196 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:23.984084    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="10a7a5ef-7ff3-42be-862f-8dd4527eecd8" containerName="main"
Sep 09 15:55:23.984196 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:23.984092    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5f94d58-8121-49d0-96ed-4890dd50b00b" containerName="main"
Sep 09 15:55:23.987183 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:23.987164    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7463-cc5045"
Sep 09 15:55:23.989379 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:23.989357    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pcz7f\""
Sep 09 15:55:23.992898 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:23.992817    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7463-cc5045"]
Sep 09 15:55:24.155212 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:24.155173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fn8n\" (UniqueName: \"kubernetes.io/projected/1c2aa256-8082-45e1-ba36-2e3930592f75-kube-api-access-6fn8n\") pod \"mt-curl-7463-cc5045\" (UID: \"1c2aa256-8082-45e1-ba36-2e3930592f75\") " pod="opendatahub/mt-curl-7463-cc5045"
Sep 09 15:55:24.256246 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:24.256150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fn8n\" (UniqueName: \"kubernetes.io/projected/1c2aa256-8082-45e1-ba36-2e3930592f75-kube-api-access-6fn8n\") pod \"mt-curl-7463-cc5045\" (UID: \"1c2aa256-8082-45e1-ba36-2e3930592f75\") " pod="opendatahub/mt-curl-7463-cc5045"
Sep 09 15:55:24.263782 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:24.263750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fn8n\" (UniqueName: \"kubernetes.io/projected/1c2aa256-8082-45e1-ba36-2e3930592f75-kube-api-access-6fn8n\") pod \"mt-curl-7463-cc5045\" (UID: \"1c2aa256-8082-45e1-ba36-2e3930592f75\") " pod="opendatahub/mt-curl-7463-cc5045"
Sep 09 15:55:24.299973 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:24.299942    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7463-cc5045"
Sep 09 15:55:24.567507 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:24.567482    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7463-cc5045"]
Sep 09 15:55:24.569532 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:55:24.569504    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c2aa256_8082_45e1_ba36_2e3930592f75.slice/crio-b7e8243ccf35566045e782dfce43aeefbbf8f684364c99ac6de423007d4274b9 WatchSource:0}: Error finding container b7e8243ccf35566045e782dfce43aeefbbf8f684364c99ac6de423007d4274b9: Status 404 returned error can't find the container with id b7e8243ccf35566045e782dfce43aeefbbf8f684364c99ac6de423007d4274b9
Sep 09 15:55:24.570776 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:24.570755    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 15:55:25.362239 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:25.362202    2578 generic.go:356] "Generic (PLEG): container finished" podID="1c2aa256-8082-45e1-ba36-2e3930592f75" containerID="384dddcebbb804f3ae4b9ecee9640bca2d863f353155fd895aaa1ce389f5c4a3" exitCode=0
Sep 09 15:55:25.362607 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:25.362258    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7463-cc5045" event={"ID":"1c2aa256-8082-45e1-ba36-2e3930592f75","Type":"ContainerDied","Data":"384dddcebbb804f3ae4b9ecee9640bca2d863f353155fd895aaa1ce389f5c4a3"}
Sep 09 15:55:25.362607 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:25.362305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7463-cc5045" event={"ID":"1c2aa256-8082-45e1-ba36-2e3930592f75","Type":"ContainerStarted","Data":"b7e8243ccf35566045e782dfce43aeefbbf8f684364c99ac6de423007d4274b9"}
Sep 09 15:55:26.498888 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:26.498853    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7463-cc5045"
Sep 09 15:55:26.517403 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:26.517381    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7463-cc5045_1c2aa256-8082-45e1-ba36-2e3930592f75/mt-curl-7463-cc5045/0.log"
Sep 09 15:55:26.546339 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:26.546306    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7463-cc5045"]
Sep 09 15:55:26.549750 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:26.549718    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7463-cc5045"]
Sep 09 15:55:26.669384 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:26.669354    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7463-30e724"]
Sep 09 15:55:26.669750 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:26.669738    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2aa256-8082-45e1-ba36-2e3930592f75" containerName="mt-curl-7463-cc5045"
Sep 09 15:55:26.669799 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:26.669751    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2aa256-8082-45e1-ba36-2e3930592f75" containerName="mt-curl-7463-cc5045"
Sep 09 15:55:26.669852 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:26.669844    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c2aa256-8082-45e1-ba36-2e3930592f75" containerName="mt-curl-7463-cc5045"
Sep 09 15:55:26.672872 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:26.672854    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7463-30e724"
Sep 09 15:55:26.678843 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:26.678819    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7463-30e724"]
Sep 09 15:55:26.678843 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:26.678831    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fn8n\" (UniqueName: \"kubernetes.io/projected/1c2aa256-8082-45e1-ba36-2e3930592f75-kube-api-access-6fn8n\") pod \"1c2aa256-8082-45e1-ba36-2e3930592f75\" (UID: \"1c2aa256-8082-45e1-ba36-2e3930592f75\") "
Sep 09 15:55:26.681657 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:26.681629    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2aa256-8082-45e1-ba36-2e3930592f75-kube-api-access-6fn8n" (OuterVolumeSpecName: "kube-api-access-6fn8n") pod "1c2aa256-8082-45e1-ba36-2e3930592f75" (UID: "1c2aa256-8082-45e1-ba36-2e3930592f75"). InnerVolumeSpecName "kube-api-access-6fn8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:55:26.779964 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:26.779933    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtt6f\" (UniqueName: \"kubernetes.io/projected/c79e2049-2c38-476a-9f76-727fe4685e0b-kube-api-access-xtt6f\") pod \"mt-curl-7463-30e724\" (UID: \"c79e2049-2c38-476a-9f76-727fe4685e0b\") " pod="opendatahub/mt-curl-7463-30e724"
Sep 09 15:55:26.780127 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:26.780063    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fn8n\" (UniqueName: \"kubernetes.io/projected/1c2aa256-8082-45e1-ba36-2e3930592f75-kube-api-access-6fn8n\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:55:26.868159 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:26.868129    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c2aa256-8082-45e1-ba36-2e3930592f75" path="/var/lib/kubelet/pods/1c2aa256-8082-45e1-ba36-2e3930592f75/volumes"
Sep 09 15:55:26.881424 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:26.881403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtt6f\" (UniqueName: \"kubernetes.io/projected/c79e2049-2c38-476a-9f76-727fe4685e0b-kube-api-access-xtt6f\") pod \"mt-curl-7463-30e724\" (UID: \"c79e2049-2c38-476a-9f76-727fe4685e0b\") " pod="opendatahub/mt-curl-7463-30e724"
Sep 09 15:55:26.888922 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:26.888902    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtt6f\" (UniqueName: \"kubernetes.io/projected/c79e2049-2c38-476a-9f76-727fe4685e0b-kube-api-access-xtt6f\") pod \"mt-curl-7463-30e724\" (UID: \"c79e2049-2c38-476a-9f76-727fe4685e0b\") " pod="opendatahub/mt-curl-7463-30e724"
Sep 09 15:55:26.995566 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:26.995478    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7463-30e724"
Sep 09 15:55:27.263209 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:27.263182    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7463-30e724"]
Sep 09 15:55:27.265090 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:55:27.265062    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc79e2049_2c38_476a_9f76_727fe4685e0b.slice/crio-3daf4ccc6545485988dc0b0bcdb97665f06303b7b0bdc603c1beb3b68d261c52 WatchSource:0}: Error finding container 3daf4ccc6545485988dc0b0bcdb97665f06303b7b0bdc603c1beb3b68d261c52: Status 404 returned error can't find the container with id 3daf4ccc6545485988dc0b0bcdb97665f06303b7b0bdc603c1beb3b68d261c52
Sep 09 15:55:27.373490 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:27.373459    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7463-cc5045"
Sep 09 15:55:27.373645 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:27.373521    2578 scope.go:117] "RemoveContainer" containerID="384dddcebbb804f3ae4b9ecee9640bca2d863f353155fd895aaa1ce389f5c4a3"
Sep 09 15:55:27.374815 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:27.374728    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7463-30e724" event={"ID":"c79e2049-2c38-476a-9f76-727fe4685e0b","Type":"ContainerStarted","Data":"3daf4ccc6545485988dc0b0bcdb97665f06303b7b0bdc603c1beb3b68d261c52"}
Sep 09 15:55:28.381413 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:28.381383    2578 generic.go:356] "Generic (PLEG): container finished" podID="c79e2049-2c38-476a-9f76-727fe4685e0b" containerID="09a52fe85271b5cc01f759ac2f3cfe68af2e245b75c7838d37dd90fa61d1a880" exitCode=0
Sep 09 15:55:28.381839 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:28.381465    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7463-30e724" event={"ID":"c79e2049-2c38-476a-9f76-727fe4685e0b","Type":"ContainerDied","Data":"09a52fe85271b5cc01f759ac2f3cfe68af2e245b75c7838d37dd90fa61d1a880"}
Sep 09 15:55:29.517518 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:29.517496    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7463-30e724"
Sep 09 15:55:29.538826 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:29.538791    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7463-30e724_c79e2049-2c38-476a-9f76-727fe4685e0b/mt-curl-7463-30e724/0.log"
Sep 09 15:55:29.565341 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:29.565308    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7463-30e724"]
Sep 09 15:55:29.569206 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:29.569183    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7463-30e724"]
Sep 09 15:55:29.606297 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:29.606264    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtt6f\" (UniqueName: \"kubernetes.io/projected/c79e2049-2c38-476a-9f76-727fe4685e0b-kube-api-access-xtt6f\") pod \"c79e2049-2c38-476a-9f76-727fe4685e0b\" (UID: \"c79e2049-2c38-476a-9f76-727fe4685e0b\") "
Sep 09 15:55:29.608503 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:29.608482    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79e2049-2c38-476a-9f76-727fe4685e0b-kube-api-access-xtt6f" (OuterVolumeSpecName: "kube-api-access-xtt6f") pod "c79e2049-2c38-476a-9f76-727fe4685e0b" (UID: "c79e2049-2c38-476a-9f76-727fe4685e0b"). InnerVolumeSpecName "kube-api-access-xtt6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:55:29.707418 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:29.707336    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtt6f\" (UniqueName: \"kubernetes.io/projected/c79e2049-2c38-476a-9f76-727fe4685e0b-kube-api-access-xtt6f\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:55:30.395018 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:30.394967    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3daf4ccc6545485988dc0b0bcdb97665f06303b7b0bdc603c1beb3b68d261c52"
Sep 09 15:55:30.395188 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:30.395022    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7463-30e724"
Sep 09 15:55:30.404779 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:55:30.404750    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7463-30e724\" is forbidden: User \"system:node:ip-10-0-134-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-183.ec2.internal' and this object" podUID="c79e2049-2c38-476a-9f76-727fe4685e0b" pod="opendatahub/mt-curl-7463-30e724"
Sep 09 15:55:30.865690 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:55:30.865610    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7463-30e724\" is forbidden: User \"system:node:ip-10-0-134-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-183.ec2.internal' and this object" podUID="c79e2049-2c38-476a-9f76-727fe4685e0b" pod="opendatahub/mt-curl-7463-30e724"
Sep 09 15:55:30.867427 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:30.867406    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c79e2049-2c38-476a-9f76-727fe4685e0b" path="/var/lib/kubelet/pods/c79e2049-2c38-476a-9f76-727fe4685e0b/volumes"
Sep 09 15:55:31.640071 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:31.640038    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"]
Sep 09 15:55:31.662936 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:31.662899    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:32.661969 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:32.661930    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:37.361097 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.361062    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"]
Sep 09 15:55:37.361474 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.361441    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c79e2049-2c38-476a-9f76-727fe4685e0b" containerName="mt-curl-7463-30e724"
Sep 09 15:55:37.361474 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.361453    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c79e2049-2c38-476a-9f76-727fe4685e0b" containerName="mt-curl-7463-30e724"
Sep 09 15:55:37.361562 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.361533    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c79e2049-2c38-476a-9f76-727fe4685e0b" containerName="mt-curl-7463-30e724"
Sep 09 15:55:37.364852 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.364833    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.367184 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.367165    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-450a51\"/\"rate-test-model-450a51-kserve-self-signed-certs\""
Sep 09 15:55:37.374473 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.374452    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"]
Sep 09 15:55:37.455300 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.455266    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:55:37.480693 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.480662    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-dshm\") pod \"rate-test-model-450a51-kserve-78558b5dc5-cxqdp\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") " pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.480838 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.480700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kh7c\" (UniqueName: \"kubernetes.io/projected/8e8eca34-fed6-4140-a054-77e33ab39f16-kube-api-access-6kh7c\") pod \"rate-test-model-450a51-kserve-78558b5dc5-cxqdp\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") " pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.480838 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.480773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-model-cache\") pod \"rate-test-model-450a51-kserve-78558b5dc5-cxqdp\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") " pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.480838 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.480798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-tmp-dir\") pod \"rate-test-model-450a51-kserve-78558b5dc5-cxqdp\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") " pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.480945 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.480846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-home\") pod \"rate-test-model-450a51-kserve-78558b5dc5-cxqdp\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") " pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.480945 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.480868    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8eca34-fed6-4140-a054-77e33ab39f16-tls-certs\") pod \"rate-test-model-450a51-kserve-78558b5dc5-cxqdp\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") " pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.581786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.581748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-dshm\") pod \"rate-test-model-450a51-kserve-78558b5dc5-cxqdp\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") " pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.581786 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.581790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kh7c\" (UniqueName: \"kubernetes.io/projected/8e8eca34-fed6-4140-a054-77e33ab39f16-kube-api-access-6kh7c\") pod \"rate-test-model-450a51-kserve-78558b5dc5-cxqdp\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") " pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.581969 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.581914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-model-cache\") pod \"rate-test-model-450a51-kserve-78558b5dc5-cxqdp\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") " pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.581969 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.581947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-tmp-dir\") pod \"rate-test-model-450a51-kserve-78558b5dc5-cxqdp\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") " pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.582120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.582029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-home\") pod \"rate-test-model-450a51-kserve-78558b5dc5-cxqdp\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") " pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.582120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.582059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8eca34-fed6-4140-a054-77e33ab39f16-tls-certs\") pod \"rate-test-model-450a51-kserve-78558b5dc5-cxqdp\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") " pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.582366 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.582345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-model-cache\") pod \"rate-test-model-450a51-kserve-78558b5dc5-cxqdp\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") " pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.582436 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.582365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-home\") pod \"rate-test-model-450a51-kserve-78558b5dc5-cxqdp\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") " pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.582436 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.582376    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-tmp-dir\") pod \"rate-test-model-450a51-kserve-78558b5dc5-cxqdp\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") " pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.584155 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.584137    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-dshm\") pod \"rate-test-model-450a51-kserve-78558b5dc5-cxqdp\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") " pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.584563 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.584546    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8eca34-fed6-4140-a054-77e33ab39f16-tls-certs\") pod \"rate-test-model-450a51-kserve-78558b5dc5-cxqdp\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") " pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.589855 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.589837    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kh7c\" (UniqueName: \"kubernetes.io/projected/8e8eca34-fed6-4140-a054-77e33ab39f16-kube-api-access-6kh7c\") pod \"rate-test-model-450a51-kserve-78558b5dc5-cxqdp\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") " pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.674194 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.674160    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:37.946656 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:37.946633    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"]
Sep 09 15:55:37.949013 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:55:37.948962    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e8eca34_fed6_4140_a054_77e33ab39f16.slice/crio-a6a95e613d13d807c2c0a48dabec2ba850edd2d3b57bc6f26fa48f6eda6ed8ab WatchSource:0}: Error finding container a6a95e613d13d807c2c0a48dabec2ba850edd2d3b57bc6f26fa48f6eda6ed8ab: Status 404 returned error can't find the container with id a6a95e613d13d807c2c0a48dabec2ba850edd2d3b57bc6f26fa48f6eda6ed8ab
Sep 09 15:55:38.441353 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:38.441317    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp" event={"ID":"8e8eca34-fed6-4140-a054-77e33ab39f16","Type":"ContainerStarted","Data":"199acde6ba88d86b05eefbd34e7cf2ed2d3114b9909baef9523c7fcbb35e38c4"}
Sep 09 15:55:38.441717 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:38.441365    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp" event={"ID":"8e8eca34-fed6-4140-a054-77e33ab39f16","Type":"ContainerStarted","Data":"a6a95e613d13d807c2c0a48dabec2ba850edd2d3b57bc6f26fa48f6eda6ed8ab"}
Sep 09 15:55:38.459466 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:38.459417    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp" podStartSLOduration=1.459400955 podStartE2EDuration="1.459400955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:55:38.457306355 +0000 UTC m=+2128.099348382" watchObservedRunningTime="2026-09-09 15:55:38.459400955 +0000 UTC m=+2128.101442944"
Sep 09 15:55:46.662140 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:46.662092    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57" podUID="7ee622e7-fe25-413b-b475-550ca9bcd3d7" containerName="main" containerID="cri-o://34204900f35bcb2936c3b539d4e9b633886afa2b1a4c63a840ec84df89aca058" gracePeriod=45
Sep 09 15:55:46.910269 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:46.910238    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:55:46.968617 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:46.968542    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-model-cache\") pod \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") "
Sep 09 15:55:46.968617 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:46.968594    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee622e7-fe25-413b-b475-550ca9bcd3d7-tls-certs\") pod \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") "
Sep 09 15:55:46.968822 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:46.968644    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-home\") pod \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") "
Sep 09 15:55:46.968822 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:46.968667    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-tmp-dir\") pod \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") "
Sep 09 15:55:46.968822 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:46.968707    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfd5q\" (UniqueName: \"kubernetes.io/projected/7ee622e7-fe25-413b-b475-550ca9bcd3d7-kube-api-access-tfd5q\") pod \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") "
Sep 09 15:55:46.969017 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:46.968812    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ee622e7-fe25-413b-b475-550ca9bcd3d7" (UID: "7ee622e7-fe25-413b-b475-550ca9bcd3d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:55:46.969017 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:46.968826    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-dshm\") pod \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\" (UID: \"7ee622e7-fe25-413b-b475-550ca9bcd3d7\") "
Sep 09 15:55:46.969017 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:46.968943    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ee622e7-fe25-413b-b475-550ca9bcd3d7" (UID: "7ee622e7-fe25-413b-b475-550ca9bcd3d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:55:46.969017 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:46.968956    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-home" (OuterVolumeSpecName: "home") pod "7ee622e7-fe25-413b-b475-550ca9bcd3d7" (UID: "7ee622e7-fe25-413b-b475-550ca9bcd3d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:55:46.969228 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:46.969179    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-model-cache\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:55:46.969228 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:46.969200    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-home\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:55:46.969228 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:46.969212    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-tmp-dir\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:55:46.971238 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:46.971213    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee622e7-fe25-413b-b475-550ca9bcd3d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ee622e7-fe25-413b-b475-550ca9bcd3d7" (UID: "7ee622e7-fe25-413b-b475-550ca9bcd3d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:55:46.971467 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:46.971450    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee622e7-fe25-413b-b475-550ca9bcd3d7-kube-api-access-tfd5q" (OuterVolumeSpecName: "kube-api-access-tfd5q") pod "7ee622e7-fe25-413b-b475-550ca9bcd3d7" (UID: "7ee622e7-fe25-413b-b475-550ca9bcd3d7"). InnerVolumeSpecName "kube-api-access-tfd5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:55:46.971516 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:46.971464    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-dshm" (OuterVolumeSpecName: "dshm") pod "7ee622e7-fe25-413b-b475-550ca9bcd3d7" (UID: "7ee622e7-fe25-413b-b475-550ca9bcd3d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:55:47.069648 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:47.069618    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfd5q\" (UniqueName: \"kubernetes.io/projected/7ee622e7-fe25-413b-b475-550ca9bcd3d7-kube-api-access-tfd5q\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:55:47.069648 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:47.069645    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee622e7-fe25-413b-b475-550ca9bcd3d7-dshm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:55:47.069648 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:47.069655    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee622e7-fe25-413b-b475-550ca9bcd3d7-tls-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:55:47.489890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:47.489852    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ee622e7-fe25-413b-b475-550ca9bcd3d7" containerID="34204900f35bcb2936c3b539d4e9b633886afa2b1a4c63a840ec84df89aca058" exitCode=0
Sep 09 15:55:47.489890 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:47.489889    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"
Sep 09 15:55:47.490120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:47.489945    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57" event={"ID":"7ee622e7-fe25-413b-b475-550ca9bcd3d7","Type":"ContainerDied","Data":"34204900f35bcb2936c3b539d4e9b633886afa2b1a4c63a840ec84df89aca058"}
Sep 09 15:55:47.490120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:47.490002    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57" event={"ID":"7ee622e7-fe25-413b-b475-550ca9bcd3d7","Type":"ContainerDied","Data":"2e4d692bf3f7ae6aba41115e1485be252b66aa470873ecf42f2afb689668ce25"}
Sep 09 15:55:47.490120 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:47.490024    2578 scope.go:117] "RemoveContainer" containerID="34204900f35bcb2936c3b539d4e9b633886afa2b1a4c63a840ec84df89aca058"
Sep 09 15:55:47.499832 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:47.499814    2578 scope.go:117] "RemoveContainer" containerID="34204900f35bcb2936c3b539d4e9b633886afa2b1a4c63a840ec84df89aca058"
Sep 09 15:55:47.500135 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:55:47.500115    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34204900f35bcb2936c3b539d4e9b633886afa2b1a4c63a840ec84df89aca058\": container with ID starting with 34204900f35bcb2936c3b539d4e9b633886afa2b1a4c63a840ec84df89aca058 not found: ID does not exist" containerID="34204900f35bcb2936c3b539d4e9b633886afa2b1a4c63a840ec84df89aca058"
Sep 09 15:55:47.500192 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:47.500143    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34204900f35bcb2936c3b539d4e9b633886afa2b1a4c63a840ec84df89aca058"} err="failed to get container status \"34204900f35bcb2936c3b539d4e9b633886afa2b1a4c63a840ec84df89aca058\": rpc error: code = NotFound desc = could not find container \"34204900f35bcb2936c3b539d4e9b633886afa2b1a4c63a840ec84df89aca058\": container with ID starting with 34204900f35bcb2936c3b539d4e9b633886afa2b1a4c63a840ec84df89aca058 not found: ID does not exist"
Sep 09 15:55:47.511006 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:47.510939    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"]
Sep 09 15:55:47.514553 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:47.514531    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-450a51/sub-test-model-450a51-kserve-f9d5dc755-vkd57"]
Sep 09 15:55:47.674794 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:47.674763    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:47.675262 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:47.674806    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:47.687661 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:47.687636    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:48.506951 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:48.506918    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:55:48.867166 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:48.867086    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee622e7-fe25-413b-b475-550ca9bcd3d7" path="/var/lib/kubelet/pods/7ee622e7-fe25-413b-b475-550ca9bcd3d7/volumes"
Sep 09 15:55:58.471241 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:55:58.471201    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:31.751901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:31.751865    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"]
Sep 09 15:56:31.775485 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:31.775447    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:32.757800 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:32.757768    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:33.247812 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:33.247773    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:33.600265 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:33.600174    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:34.037226 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:34.037200    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:36.438770 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:36.438734    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:36.537514 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:36.537487    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:37.541299 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.541263    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"]
Sep 09 15:56:37.541924 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.541902    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee622e7-fe25-413b-b475-550ca9bcd3d7" containerName="main"
Sep 09 15:56:37.541989 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.541932    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee622e7-fe25-413b-b475-550ca9bcd3d7" containerName="main"
Sep 09 15:56:37.542090 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.542077    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee622e7-fe25-413b-b475-550ca9bcd3d7" containerName="main"
Sep 09 15:56:37.545745 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.545727    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"
Sep 09 15:56:37.548025 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.548003    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-1218fe-serving-cert\""
Sep 09 15:56:37.557376 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.557331    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"]
Sep 09 15:56:37.573486 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.573431    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:37.631488 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.631456    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:37.663053 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.663023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64c5e348-bf04-44e1-8427-d01a2e5ad69a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z\" (UID: \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"
Sep 09 15:56:37.663205 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.663068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c5e348-bf04-44e1-8427-d01a2e5ad69a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z\" (UID: \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"
Sep 09 15:56:37.663205 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.663140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64c5e348-bf04-44e1-8427-d01a2e5ad69a-metrics-certs\") pod \"maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z\" (UID: \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"
Sep 09 15:56:37.663336 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.663197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnpc2\" (UniqueName: \"kubernetes.io/projected/64c5e348-bf04-44e1-8427-d01a2e5ad69a-kube-api-access-qnpc2\") pod \"maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z\" (UID: \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"
Sep 09 15:56:37.764267 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.764239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64c5e348-bf04-44e1-8427-d01a2e5ad69a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z\" (UID: \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"
Sep 09 15:56:37.764445 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.764282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c5e348-bf04-44e1-8427-d01a2e5ad69a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z\" (UID: \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"
Sep 09 15:56:37.764445 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.764338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64c5e348-bf04-44e1-8427-d01a2e5ad69a-metrics-certs\") pod \"maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z\" (UID: \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"
Sep 09 15:56:37.764445 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.764356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnpc2\" (UniqueName: \"kubernetes.io/projected/64c5e348-bf04-44e1-8427-d01a2e5ad69a-kube-api-access-qnpc2\") pod \"maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z\" (UID: \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"
Sep 09 15:56:37.765046 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.765019    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c5e348-bf04-44e1-8427-d01a2e5ad69a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z\" (UID: \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"
Sep 09 15:56:37.766997 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.766961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64c5e348-bf04-44e1-8427-d01a2e5ad69a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z\" (UID: \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"
Sep 09 15:56:37.767102 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.767071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64c5e348-bf04-44e1-8427-d01a2e5ad69a-metrics-certs\") pod \"maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z\" (UID: \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"
Sep 09 15:56:37.772520 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.772490    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnpc2\" (UniqueName: \"kubernetes.io/projected/64c5e348-bf04-44e1-8427-d01a2e5ad69a-kube-api-access-qnpc2\") pod \"maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z\" (UID: \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"
Sep 09 15:56:37.867057 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:37.866961    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"
Sep 09 15:56:38.144817 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:38.144776    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"]
Sep 09 15:56:38.148801 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:56:38.148770    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c5e348_bf04_44e1_8427_d01a2e5ad69a.slice/crio-b78fc549c4a68504351dbcd88e36622c2e88241985f2f6684b356e1077bb1c94 WatchSource:0}: Error finding container b78fc549c4a68504351dbcd88e36622c2e88241985f2f6684b356e1077bb1c94: Status 404 returned error can't find the container with id b78fc549c4a68504351dbcd88e36622c2e88241985f2f6684b356e1077bb1c94
Sep 09 15:56:38.750451 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:38.750425    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z" event={"ID":"64c5e348-bf04-44e1-8427-d01a2e5ad69a","Type":"ContainerStarted","Data":"b78fc549c4a68504351dbcd88e36622c2e88241985f2f6684b356e1077bb1c94"}
Sep 09 15:56:39.757312 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:39.757275    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z" event={"ID":"64c5e348-bf04-44e1-8427-d01a2e5ad69a","Type":"ContainerStarted","Data":"86c8300c9da7ec56ef965b37b7ee22ea1ad183861e0552615be31d065aa0b0d2"}
Sep 09 15:56:39.757764 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:39.757403    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"
Sep 09 15:56:39.775694 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:39.775646    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z" podStartSLOduration=2.1978187 podStartE2EDuration="2.775632072s" totalImagesPullingTime="577.813372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:56:39.773411058 +0000 UTC m=+2189.415453048" watchObservedRunningTime="2026-09-09 15:56:39.775632072 +0000 UTC m=+2189.417674061"
Sep 09 15:56:40.844287 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:40.844255    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:40.898408 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:40.898354    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:56:45.768397 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:45.768362    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"
Sep 09 15:56:46.772371 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:46.772326    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp" podUID="8e8eca34-fed6-4140-a054-77e33ab39f16" containerName="main" containerID="cri-o://199acde6ba88d86b05eefbd34e7cf2ed2d3114b9909baef9523c7fcbb35e38c4" gracePeriod=45
Sep 09 15:56:47.026238 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.026172    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:56:47.050335 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.050309    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kh7c\" (UniqueName: \"kubernetes.io/projected/8e8eca34-fed6-4140-a054-77e33ab39f16-kube-api-access-6kh7c\") pod \"8e8eca34-fed6-4140-a054-77e33ab39f16\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") "
Sep 09 15:56:47.050501 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.050355    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-dshm\") pod \"8e8eca34-fed6-4140-a054-77e33ab39f16\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") "
Sep 09 15:56:47.050501 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.050391    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8eca34-fed6-4140-a054-77e33ab39f16-tls-certs\") pod \"8e8eca34-fed6-4140-a054-77e33ab39f16\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") "
Sep 09 15:56:47.050501 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.050429    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-model-cache\") pod \"8e8eca34-fed6-4140-a054-77e33ab39f16\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") "
Sep 09 15:56:47.050501 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.050473    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-home\") pod \"8e8eca34-fed6-4140-a054-77e33ab39f16\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") "
Sep 09 15:56:47.050501 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.050501    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-tmp-dir\") pod \"8e8eca34-fed6-4140-a054-77e33ab39f16\" (UID: \"8e8eca34-fed6-4140-a054-77e33ab39f16\") "
Sep 09 15:56:47.050792 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.050762    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e8eca34-fed6-4140-a054-77e33ab39f16" (UID: "8e8eca34-fed6-4140-a054-77e33ab39f16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:56:47.050850 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.050788    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-home" (OuterVolumeSpecName: "home") pod "8e8eca34-fed6-4140-a054-77e33ab39f16" (UID: "8e8eca34-fed6-4140-a054-77e33ab39f16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:56:47.050930 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.050917    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e8eca34-fed6-4140-a054-77e33ab39f16" (UID: "8e8eca34-fed6-4140-a054-77e33ab39f16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:56:47.052772 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.052747    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e8eca34-fed6-4140-a054-77e33ab39f16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e8eca34-fed6-4140-a054-77e33ab39f16" (UID: "8e8eca34-fed6-4140-a054-77e33ab39f16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:56:47.053094 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.053071    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-dshm" (OuterVolumeSpecName: "dshm") pod "8e8eca34-fed6-4140-a054-77e33ab39f16" (UID: "8e8eca34-fed6-4140-a054-77e33ab39f16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:56:47.053185 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.053093    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8eca34-fed6-4140-a054-77e33ab39f16-kube-api-access-6kh7c" (OuterVolumeSpecName: "kube-api-access-6kh7c") pod "8e8eca34-fed6-4140-a054-77e33ab39f16" (UID: "8e8eca34-fed6-4140-a054-77e33ab39f16"). InnerVolumeSpecName "kube-api-access-6kh7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:56:47.151559 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.151527    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-tmp-dir\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:56:47.151559 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.151555    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kh7c\" (UniqueName: \"kubernetes.io/projected/8e8eca34-fed6-4140-a054-77e33ab39f16-kube-api-access-6kh7c\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:56:47.151559 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.151564    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-dshm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:56:47.151778 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.151573    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8eca34-fed6-4140-a054-77e33ab39f16-tls-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:56:47.151778 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.151583    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-model-cache\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:56:47.151778 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.151590    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8eca34-fed6-4140-a054-77e33ab39f16-home\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:56:47.798503 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.798470    2578 generic.go:356] "Generic (PLEG): container finished" podID="8e8eca34-fed6-4140-a054-77e33ab39f16" containerID="199acde6ba88d86b05eefbd34e7cf2ed2d3114b9909baef9523c7fcbb35e38c4" exitCode=0
Sep 09 15:56:47.799010 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.798514    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"
Sep 09 15:56:47.799010 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.798518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp" event={"ID":"8e8eca34-fed6-4140-a054-77e33ab39f16","Type":"ContainerDied","Data":"199acde6ba88d86b05eefbd34e7cf2ed2d3114b9909baef9523c7fcbb35e38c4"}
Sep 09 15:56:47.799010 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.798649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp" event={"ID":"8e8eca34-fed6-4140-a054-77e33ab39f16","Type":"ContainerDied","Data":"a6a95e613d13d807c2c0a48dabec2ba850edd2d3b57bc6f26fa48f6eda6ed8ab"}
Sep 09 15:56:47.799010 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.798673    2578 scope.go:117] "RemoveContainer" containerID="199acde6ba88d86b05eefbd34e7cf2ed2d3114b9909baef9523c7fcbb35e38c4"
Sep 09 15:56:47.807961 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.807946    2578 scope.go:117] "RemoveContainer" containerID="199acde6ba88d86b05eefbd34e7cf2ed2d3114b9909baef9523c7fcbb35e38c4"
Sep 09 15:56:47.808277 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:56:47.808260    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"199acde6ba88d86b05eefbd34e7cf2ed2d3114b9909baef9523c7fcbb35e38c4\": container with ID starting with 199acde6ba88d86b05eefbd34e7cf2ed2d3114b9909baef9523c7fcbb35e38c4 not found: ID does not exist" containerID="199acde6ba88d86b05eefbd34e7cf2ed2d3114b9909baef9523c7fcbb35e38c4"
Sep 09 15:56:47.808353 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.808285    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"199acde6ba88d86b05eefbd34e7cf2ed2d3114b9909baef9523c7fcbb35e38c4"} err="failed to get container status \"199acde6ba88d86b05eefbd34e7cf2ed2d3114b9909baef9523c7fcbb35e38c4\": rpc error: code = NotFound desc = could not find container \"199acde6ba88d86b05eefbd34e7cf2ed2d3114b9909baef9523c7fcbb35e38c4\": container with ID starting with 199acde6ba88d86b05eefbd34e7cf2ed2d3114b9909baef9523c7fcbb35e38c4 not found: ID does not exist"
Sep 09 15:56:47.819746 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.819723    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"]
Sep 09 15:56:47.824169 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:47.824149    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-450a51/rate-test-model-450a51-kserve-78558b5dc5-cxqdp"]
Sep 09 15:56:48.867769 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:48.867735    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e8eca34-fed6-4140-a054-77e33ab39f16" path="/var/lib/kubelet/pods/8e8eca34-fed6-4140-a054-77e33ab39f16/volumes"
Sep 09 15:56:52.298110 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:56:52.298073    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:01.567223 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:01.567184    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:02.321342 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:02.321306    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:02.681728 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:02.681690    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:02.915637 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:02.915608    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:06.021492 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:06.021463    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:06.121595 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:06.121564    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:07.143745 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.143709    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"]
Sep 09 15:57:07.144413 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.144388    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e8eca34-fed6-4140-a054-77e33ab39f16" containerName="main"
Sep 09 15:57:07.144413 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.144413    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e8eca34-fed6-4140-a054-77e33ab39f16" containerName="main"
Sep 09 15:57:07.144621 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.144540    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e8eca34-fed6-4140-a054-77e33ab39f16" containerName="main"
Sep 09 15:57:07.149090 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.149068    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 15:57:07.151550 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.151525    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a2b442-serving-cert\""
Sep 09 15:57:07.159158 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.159135    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"]
Sep 09 15:57:07.187003 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.186949    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:07.234465 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.234431    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wb5j\" (UniqueName: \"kubernetes.io/projected/18557905-9aed-41eb-bf3e-8f58d93eb8f2-kube-api-access-4wb5j\") pod \"maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7\" (UID: \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 15:57:07.234646 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.234520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18557905-9aed-41eb-bf3e-8f58d93eb8f2-metrics-certs\") pod \"maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7\" (UID: \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 15:57:07.234646 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.234558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18557905-9aed-41eb-bf3e-8f58d93eb8f2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7\" (UID: \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 15:57:07.234646 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.234610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18557905-9aed-41eb-bf3e-8f58d93eb8f2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7\" (UID: \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 15:57:07.288196 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.288158    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:07.336027 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.336000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18557905-9aed-41eb-bf3e-8f58d93eb8f2-metrics-certs\") pod \"maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7\" (UID: \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 15:57:07.336133 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.336055    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18557905-9aed-41eb-bf3e-8f58d93eb8f2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7\" (UID: \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 15:57:07.336133 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.336097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18557905-9aed-41eb-bf3e-8f58d93eb8f2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7\" (UID: \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 15:57:07.336133 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.336119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wb5j\" (UniqueName: \"kubernetes.io/projected/18557905-9aed-41eb-bf3e-8f58d93eb8f2-kube-api-access-4wb5j\") pod \"maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7\" (UID: \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 15:57:07.336273 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:57:07.336231    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-serving-cert: secret "maas-api-e2e-ipp-b-a2b442-serving-cert" not found
Sep 09 15:57:07.336315 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:57:07.336301    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18557905-9aed-41eb-bf3e-8f58d93eb8f2-maas-api-tls podName:18557905-9aed-41eb-bf3e-8f58d93eb8f2 nodeName:}" failed. No retries permitted until 2026-09-09 15:57:07.836281103 +0000 UTC m=+2217.478323075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/18557905-9aed-41eb-bf3e-8f58d93eb8f2-maas-api-tls") pod "maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7" (UID: "18557905-9aed-41eb-bf3e-8f58d93eb8f2") : secret "maas-api-e2e-ipp-b-a2b442-serving-cert" not found
Sep 09 15:57:07.336774 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.336750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18557905-9aed-41eb-bf3e-8f58d93eb8f2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7\" (UID: \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 15:57:07.338596 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.338569    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18557905-9aed-41eb-bf3e-8f58d93eb8f2-metrics-certs\") pod \"maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7\" (UID: \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 15:57:07.344365 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.344341    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wb5j\" (UniqueName: \"kubernetes.io/projected/18557905-9aed-41eb-bf3e-8f58d93eb8f2-kube-api-access-4wb5j\") pod \"maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7\" (UID: \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 15:57:07.516713 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.516627    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:07.663870 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.663840    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:07.757257 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.757224    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:07.842512 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.842425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18557905-9aed-41eb-bf3e-8f58d93eb8f2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7\" (UID: \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 15:57:07.845141 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:07.845117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18557905-9aed-41eb-bf3e-8f58d93eb8f2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7\" (UID: \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 15:57:08.062620 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:08.062589    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 15:57:08.333912 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:08.333881    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"]
Sep 09 15:57:08.335890 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:57:08.335859    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18557905_9aed_41eb_bf3e_8f58d93eb8f2.slice/crio-c2c5dc9f52aedbba04389e583739c01a65442406996775fc4ef7c1812b1688f9 WatchSource:0}: Error finding container c2c5dc9f52aedbba04389e583739c01a65442406996775fc4ef7c1812b1688f9: Status 404 returned error can't find the container with id c2c5dc9f52aedbba04389e583739c01a65442406996775fc4ef7c1812b1688f9
Sep 09 15:57:08.909078 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:08.909045    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7" event={"ID":"18557905-9aed-41eb-bf3e-8f58d93eb8f2","Type":"ContainerStarted","Data":"c2c5dc9f52aedbba04389e583739c01a65442406996775fc4ef7c1812b1688f9"}
Sep 09 15:57:09.065407 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:09.065379    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:10.921649 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:10.921613    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7" event={"ID":"18557905-9aed-41eb-bf3e-8f58d93eb8f2","Type":"ContainerStarted","Data":"e141e8db547fc18544e58c8648f8859c6a08e1d3a2dd32700d511ef53cced9c1"}
Sep 09 15:57:10.922093 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:10.921752    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 15:57:10.938343 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:10.938301    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7" podStartSLOduration=2.36897225 podStartE2EDuration="3.938286164s" totalImagesPullingTime="1.569313914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:57:10.937740696 +0000 UTC m=+2220.579782690" watchObservedRunningTime="2026-09-09 15:57:10.938286164 +0000 UTC m=+2220.580328154"
Sep 09 15:57:16.933233 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:16.933203    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 15:57:28.044960 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:28.044921    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:31.091220 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:31.091189    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:36.335004 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:36.334899    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-qf4s4_c743a297-9127-4f66-8153-8a00d9327040/payload-processing/0.log"
Sep 09 15:57:38.071841 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:38.071806    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:44.548324 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.548276    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"]
Sep 09 15:57:44.553220 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.553191    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.556703 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.556672    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1218fe\"/\"openshift-service-ca.crt\""
Sep 09 15:57:44.556703 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.556692    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1218fe\"/\"ipp-route-1218fe-kserve-self-signed-certs\""
Sep 09 15:57:44.556926 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.556728    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1218fe\"/\"default-dockercfg-qmd7c\""
Sep 09 15:57:44.556926 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.556672    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1218fe\"/\"kube-root-ca.crt\""
Sep 09 15:57:44.559646 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.559621    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"]
Sep 09 15:57:44.575233 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.575204    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:44.597337 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.597296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pq6p\" (UniqueName: \"kubernetes.io/projected/de66609a-367c-4f7b-8d89-8467a6af4012-kube-api-access-4pq6p\") pod \"ipp-route-1218fe-kserve-c89f4f655-fw94c\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") " pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.597454 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.597354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-home\") pod \"ipp-route-1218fe-kserve-c89f4f655-fw94c\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") " pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.597528 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.597452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-model-cache\") pod \"ipp-route-1218fe-kserve-c89f4f655-fw94c\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") " pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.597528 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.597482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de66609a-367c-4f7b-8d89-8467a6af4012-tls-certs\") pod \"ipp-route-1218fe-kserve-c89f4f655-fw94c\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") " pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.597528 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.597524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-tmp-dir\") pod \"ipp-route-1218fe-kserve-c89f4f655-fw94c\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") " pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.597636 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.597619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-dshm\") pod \"ipp-route-1218fe-kserve-c89f4f655-fw94c\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") " pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.698657 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.698621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-dshm\") pod \"ipp-route-1218fe-kserve-c89f4f655-fw94c\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") " pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.698833 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.698664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pq6p\" (UniqueName: \"kubernetes.io/projected/de66609a-367c-4f7b-8d89-8467a6af4012-kube-api-access-4pq6p\") pod \"ipp-route-1218fe-kserve-c89f4f655-fw94c\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") " pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.698833 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.698691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-home\") pod \"ipp-route-1218fe-kserve-c89f4f655-fw94c\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") " pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.698833 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.698731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-model-cache\") pod \"ipp-route-1218fe-kserve-c89f4f655-fw94c\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") " pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.698833 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.698749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de66609a-367c-4f7b-8d89-8467a6af4012-tls-certs\") pod \"ipp-route-1218fe-kserve-c89f4f655-fw94c\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") " pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.698833 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.698773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-tmp-dir\") pod \"ipp-route-1218fe-kserve-c89f4f655-fw94c\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") " pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.699157 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.699135    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-model-cache\") pod \"ipp-route-1218fe-kserve-c89f4f655-fw94c\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") " pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.699213 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.699193    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-tmp-dir\") pod \"ipp-route-1218fe-kserve-c89f4f655-fw94c\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") " pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.699213 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.699204    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-home\") pod \"ipp-route-1218fe-kserve-c89f4f655-fw94c\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") " pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.701096 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.701076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-dshm\") pod \"ipp-route-1218fe-kserve-c89f4f655-fw94c\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") " pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.701361 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.701345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de66609a-367c-4f7b-8d89-8467a6af4012-tls-certs\") pod \"ipp-route-1218fe-kserve-c89f4f655-fw94c\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") " pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.706848 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.706825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pq6p\" (UniqueName: \"kubernetes.io/projected/de66609a-367c-4f7b-8d89-8467a6af4012-kube-api-access-4pq6p\") pod \"ipp-route-1218fe-kserve-c89f4f655-fw94c\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") " pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:44.871600 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:44.871521    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:45.145694 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:45.145668    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"]
Sep 09 15:57:45.147878 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:57:45.147847    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde66609a_367c_4f7b_8d89_8467a6af4012.slice/crio-83b1c3cf596759442a5e12d2ee06410babb49698bf7a6e7897e6370d7f1e69b9 WatchSource:0}: Error finding container 83b1c3cf596759442a5e12d2ee06410babb49698bf7a6e7897e6370d7f1e69b9: Status 404 returned error can't find the container with id 83b1c3cf596759442a5e12d2ee06410babb49698bf7a6e7897e6370d7f1e69b9
Sep 09 15:57:46.109111 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:46.109064    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c" event={"ID":"de66609a-367c-4f7b-8d89-8467a6af4012","Type":"ContainerStarted","Data":"4e335f37333b241e3a250bb6d6df1eee89f078be7a1274dc96116e1181126d8d"}
Sep 09 15:57:46.109111 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:46.109116    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c" event={"ID":"de66609a-367c-4f7b-8d89-8467a6af4012","Type":"ContainerStarted","Data":"83b1c3cf596759442a5e12d2ee06410babb49698bf7a6e7897e6370d7f1e69b9"}
Sep 09 15:57:46.125858 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:46.125809    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c" podStartSLOduration=2.125790989 podStartE2EDuration="2.125790989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:57:46.125101607 +0000 UTC m=+2255.767143597" watchObservedRunningTime="2026-09-09 15:57:46.125790989 +0000 UTC m=+2255.767832990"
Sep 09 15:57:47.659183 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:47.659147    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:48.338928 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:48.338891    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:57:54.871842 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:54.871808    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:54.871842 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:54.871848    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:54.885000 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:54.884957    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:57:55.166414 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:57:55.166387    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:58:07.465754 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:07.465718    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-qf4s4_c743a297-9127-4f66-8153-8a00d9327040/payload-processing/0.log"
Sep 09 15:58:07.728445 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:07.728359    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:07.886638 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:07.886610    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:08.222779 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:08.222746    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:08.367992 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:08.367945    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:11.756114 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:11.756082    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:11.854706 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:11.854680    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:12.996778 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:12.996736    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"]
Sep 09 15:58:13.000846 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.000825    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:13.003089 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.003068    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-c6ae87-serving-cert\""
Sep 09 15:58:13.010156 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.010133    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"]
Sep 09 15:58:13.024225 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.024199    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:13.064766 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.064727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8\" (UID: \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:13.064929 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.064792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8\" (UID: \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:13.064929 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.064838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8\" (UID: \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:13.064929 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.064862    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgjlm\" (UniqueName: \"kubernetes.io/projected/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-kube-api-access-mgjlm\") pod \"maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8\" (UID: \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:13.123595 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.123560    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:13.166069 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.166033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8\" (UID: \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:13.166239 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.166108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8\" (UID: \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:13.166239 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.166160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8\" (UID: \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:13.166239 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.166177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgjlm\" (UniqueName: \"kubernetes.io/projected/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-kube-api-access-mgjlm\") pod \"maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8\" (UID: \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:13.166239 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:58:13.166204    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-serving-cert: secret "maas-api-e2e-ipp-cleanup-c6ae87-serving-cert" not found
Sep 09 15:58:13.166413 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:58:13.166294    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-maas-api-tls podName:b752b7c7-1dc5-47b4-ae90-4a619c4038b4 nodeName:}" failed. No retries permitted until 2026-09-09 15:58:13.666273074 +0000 UTC m=+2283.308315042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8" (UID: "b752b7c7-1dc5-47b4-ae90-4a619c4038b4") : secret "maas-api-e2e-ipp-cleanup-c6ae87-serving-cert" not found
Sep 09 15:58:13.167048 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.167010    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8\" (UID: \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:13.168696 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.168678    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8\" (UID: \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:13.174930 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.174904    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgjlm\" (UniqueName: \"kubernetes.io/projected/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-kube-api-access-mgjlm\") pod \"maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8\" (UID: \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:13.227316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.227281    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:13.484661 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.484622    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:13.577476 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.577445    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:13.673085 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.673043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8\" (UID: \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:13.675749 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.675728    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8\" (UID: \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:13.914002 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:13.913942    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:14.186033 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:14.186002    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"]
Sep 09 15:58:14.188240 ip-10-0-134-183 kubenswrapper[2578]: W0909 15:58:14.188200    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb752b7c7_1dc5_47b4_ae90_4a619c4038b4.slice/crio-7a62f1830e134a845586338e1f670f21815525ac13f450acd7ff5e925b1f0d81 WatchSource:0}: Error finding container 7a62f1830e134a845586338e1f670f21815525ac13f450acd7ff5e925b1f0d81: Status 404 returned error can't find the container with id 7a62f1830e134a845586338e1f670f21815525ac13f450acd7ff5e925b1f0d81
Sep 09 15:58:14.254055 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:14.254019    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8" event={"ID":"b752b7c7-1dc5-47b4-ae90-4a619c4038b4","Type":"ContainerStarted","Data":"7a62f1830e134a845586338e1f670f21815525ac13f450acd7ff5e925b1f0d81"}
Sep 09 15:58:19.282834 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:19.282801    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8" event={"ID":"b752b7c7-1dc5-47b4-ae90-4a619c4038b4","Type":"ContainerStarted","Data":"bb8ff51161cc976f04a14082eb04e127a79ff555754566327f2d9d6527131856"}
Sep 09 15:58:19.283250 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:19.282875    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:19.300750 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:19.300704    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8" podStartSLOduration=3.010686766 podStartE2EDuration="7.300690013s" totalImagesPullingTime="4.290003247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 15:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:58:19.299622588 +0000 UTC m=+2288.941664574" watchObservedRunningTime="2026-09-09 15:58:19.300690013 +0000 UTC m=+2288.942732001"
Sep 09 15:58:25.294073 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:25.294044    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:36.914441 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:36.914407    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:43.667741 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:43.667701    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"]
Sep 09 15:58:43.668312 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:43.667994    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8" podUID="b752b7c7-1dc5-47b4-ae90-4a619c4038b4" containerName="maas-api" containerID="cri-o://bb8ff51161cc976f04a14082eb04e127a79ff555754566327f2d9d6527131856" gracePeriod=30
Sep 09 15:58:43.692310 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:43.692277    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:43.792885 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:43.792859    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:43.873099 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:43.873068    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:43.914435 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:43.914408    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:44.086732 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.086643    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-metrics-certs\") pod \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\" (UID: \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\") "
Sep 09 15:58:44.086732 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.086716    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-maas-api-tls\") pod \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\" (UID: \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\") "
Sep 09 15:58:44.086928 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.086825    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-openshift-service-ca\") pod \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\" (UID: \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\") "
Sep 09 15:58:44.086928 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.086874    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgjlm\" (UniqueName: \"kubernetes.io/projected/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-kube-api-access-mgjlm\") pod \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\" (UID: \"b752b7c7-1dc5-47b4-ae90-4a619c4038b4\") "
Sep 09 15:58:44.087222 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.087193    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b752b7c7-1dc5-47b4-ae90-4a619c4038b4" (UID: "b752b7c7-1dc5-47b4-ae90-4a619c4038b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:58:44.089113 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.089082    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b752b7c7-1dc5-47b4-ae90-4a619c4038b4" (UID: "b752b7c7-1dc5-47b4-ae90-4a619c4038b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:58:44.089213 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.089176    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b752b7c7-1dc5-47b4-ae90-4a619c4038b4" (UID: "b752b7c7-1dc5-47b4-ae90-4a619c4038b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:58:44.089213 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.089194    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-kube-api-access-mgjlm" (OuterVolumeSpecName: "kube-api-access-mgjlm") pod "b752b7c7-1dc5-47b4-ae90-4a619c4038b4" (UID: "b752b7c7-1dc5-47b4-ae90-4a619c4038b4"). InnerVolumeSpecName "kube-api-access-mgjlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:58:44.188459 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.188412    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:58:44.188459 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.188449    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-maas-api-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:58:44.188459 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.188462    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-openshift-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:58:44.188704 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.188474    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgjlm\" (UniqueName: \"kubernetes.io/projected/b752b7c7-1dc5-47b4-ae90-4a619c4038b4-kube-api-access-mgjlm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:58:44.418923 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.418885    2578 generic.go:356] "Generic (PLEG): container finished" podID="b752b7c7-1dc5-47b4-ae90-4a619c4038b4" containerID="bb8ff51161cc976f04a14082eb04e127a79ff555754566327f2d9d6527131856" exitCode=0
Sep 09 15:58:44.419124 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.418937    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"
Sep 09 15:58:44.419124 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.419000    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8" event={"ID":"b752b7c7-1dc5-47b4-ae90-4a619c4038b4","Type":"ContainerDied","Data":"bb8ff51161cc976f04a14082eb04e127a79ff555754566327f2d9d6527131856"}
Sep 09 15:58:44.419124 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.419055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8" event={"ID":"b752b7c7-1dc5-47b4-ae90-4a619c4038b4","Type":"ContainerDied","Data":"7a62f1830e134a845586338e1f670f21815525ac13f450acd7ff5e925b1f0d81"}
Sep 09 15:58:44.419124 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.419078    2578 scope.go:117] "RemoveContainer" containerID="bb8ff51161cc976f04a14082eb04e127a79ff555754566327f2d9d6527131856"
Sep 09 15:58:44.429504 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.429482    2578 scope.go:117] "RemoveContainer" containerID="bb8ff51161cc976f04a14082eb04e127a79ff555754566327f2d9d6527131856"
Sep 09 15:58:44.429785 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:58:44.429766    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb8ff51161cc976f04a14082eb04e127a79ff555754566327f2d9d6527131856\": container with ID starting with bb8ff51161cc976f04a14082eb04e127a79ff555754566327f2d9d6527131856 not found: ID does not exist" containerID="bb8ff51161cc976f04a14082eb04e127a79ff555754566327f2d9d6527131856"
Sep 09 15:58:44.429837 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.429795    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb8ff51161cc976f04a14082eb04e127a79ff555754566327f2d9d6527131856"} err="failed to get container status \"bb8ff51161cc976f04a14082eb04e127a79ff555754566327f2d9d6527131856\": rpc error: code = NotFound desc = could not find container \"bb8ff51161cc976f04a14082eb04e127a79ff555754566327f2d9d6527131856\": container with ID starting with bb8ff51161cc976f04a14082eb04e127a79ff555754566327f2d9d6527131856 not found: ID does not exist"
Sep 09 15:58:44.440479 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.440450    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"]
Sep 09 15:58:44.443795 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.443765    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c6ae87-85cd4b7dbc-8sbw8"]
Sep 09 15:58:44.867922 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:44.867840    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b752b7c7-1dc5-47b4-ae90-4a619c4038b4" path="/var/lib/kubelet/pods/b752b7c7-1dc5-47b4-ae90-4a619c4038b4/volumes"
Sep 09 15:58:47.278382 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:47.278344    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:47.343032 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:47.343002    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:49.980000 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:49.979953    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:50.081042 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:50.081008    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:50.282831 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:50.282754    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:58:55.060484 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:58:55.060451    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:59:02.141316 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:02.141239    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:59:09.858779 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:09.858748    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"]
Sep 09 15:59:09.859193 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:09.858960    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z" podUID="64c5e348-bf04-44e1-8427-d01a2e5ad69a" containerName="maas-api" containerID="cri-o://86c8300c9da7ec56ef965b37b7ee22ea1ad183861e0552615be31d065aa0b0d2" gracePeriod=30
Sep 09 15:59:09.890158 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:09.890132    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:59:09.988476 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:09.988436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:59:10.091934 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.091795    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:59:10.113582 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.113517    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"
Sep 09 15:59:10.244229 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.244200    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64c5e348-bf04-44e1-8427-d01a2e5ad69a-maas-api-tls\") pod \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\" (UID: \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\") "
Sep 09 15:59:10.244428 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.244295    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c5e348-bf04-44e1-8427-d01a2e5ad69a-openshift-service-ca\") pod \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\" (UID: \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\") "
Sep 09 15:59:10.244428 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.244394    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnpc2\" (UniqueName: \"kubernetes.io/projected/64c5e348-bf04-44e1-8427-d01a2e5ad69a-kube-api-access-qnpc2\") pod \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\" (UID: \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\") "
Sep 09 15:59:10.244559 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.244426    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64c5e348-bf04-44e1-8427-d01a2e5ad69a-metrics-certs\") pod \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\" (UID: \"64c5e348-bf04-44e1-8427-d01a2e5ad69a\") "
Sep 09 15:59:10.244693 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.244667    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c5e348-bf04-44e1-8427-d01a2e5ad69a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "64c5e348-bf04-44e1-8427-d01a2e5ad69a" (UID: "64c5e348-bf04-44e1-8427-d01a2e5ad69a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 15:59:10.244797 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.244785    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c5e348-bf04-44e1-8427-d01a2e5ad69a-openshift-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:59:10.246531 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.246507    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64c5e348-bf04-44e1-8427-d01a2e5ad69a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "64c5e348-bf04-44e1-8427-d01a2e5ad69a" (UID: "64c5e348-bf04-44e1-8427-d01a2e5ad69a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:59:10.246681 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.246658    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64c5e348-bf04-44e1-8427-d01a2e5ad69a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "64c5e348-bf04-44e1-8427-d01a2e5ad69a" (UID: "64c5e348-bf04-44e1-8427-d01a2e5ad69a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:59:10.246785 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.246727    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c5e348-bf04-44e1-8427-d01a2e5ad69a-kube-api-access-qnpc2" (OuterVolumeSpecName: "kube-api-access-qnpc2") pod "64c5e348-bf04-44e1-8427-d01a2e5ad69a" (UID: "64c5e348-bf04-44e1-8427-d01a2e5ad69a"). InnerVolumeSpecName "kube-api-access-qnpc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:59:10.252216 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.252195    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:59:10.345849 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.345814    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnpc2\" (UniqueName: \"kubernetes.io/projected/64c5e348-bf04-44e1-8427-d01a2e5ad69a-kube-api-access-qnpc2\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:59:10.345849 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.345849    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64c5e348-bf04-44e1-8427-d01a2e5ad69a-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:59:10.346069 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.345858    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64c5e348-bf04-44e1-8427-d01a2e5ad69a-maas-api-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:59:10.566997 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.566945    2578 generic.go:356] "Generic (PLEG): container finished" podID="64c5e348-bf04-44e1-8427-d01a2e5ad69a" containerID="86c8300c9da7ec56ef965b37b7ee22ea1ad183861e0552615be31d065aa0b0d2" exitCode=0
Sep 09 15:59:10.567182 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.567005    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"
Sep 09 15:59:10.567182 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.567021    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z" event={"ID":"64c5e348-bf04-44e1-8427-d01a2e5ad69a","Type":"ContainerDied","Data":"86c8300c9da7ec56ef965b37b7ee22ea1ad183861e0552615be31d065aa0b0d2"}
Sep 09 15:59:10.567182 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.567065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z" event={"ID":"64c5e348-bf04-44e1-8427-d01a2e5ad69a","Type":"ContainerDied","Data":"b78fc549c4a68504351dbcd88e36622c2e88241985f2f6684b356e1077bb1c94"}
Sep 09 15:59:10.567182 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.567086    2578 scope.go:117] "RemoveContainer" containerID="86c8300c9da7ec56ef965b37b7ee22ea1ad183861e0552615be31d065aa0b0d2"
Sep 09 15:59:10.576879 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.576859    2578 scope.go:117] "RemoveContainer" containerID="86c8300c9da7ec56ef965b37b7ee22ea1ad183861e0552615be31d065aa0b0d2"
Sep 09 15:59:10.577169 ip-10-0-134-183 kubenswrapper[2578]: E0909 15:59:10.577147    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c8300c9da7ec56ef965b37b7ee22ea1ad183861e0552615be31d065aa0b0d2\": container with ID starting with 86c8300c9da7ec56ef965b37b7ee22ea1ad183861e0552615be31d065aa0b0d2 not found: ID does not exist" containerID="86c8300c9da7ec56ef965b37b7ee22ea1ad183861e0552615be31d065aa0b0d2"
Sep 09 15:59:10.577263 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.577176    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c8300c9da7ec56ef965b37b7ee22ea1ad183861e0552615be31d065aa0b0d2"} err="failed to get container status \"86c8300c9da7ec56ef965b37b7ee22ea1ad183861e0552615be31d065aa0b0d2\": rpc error: code = NotFound desc = could not find container \"86c8300c9da7ec56ef965b37b7ee22ea1ad183861e0552615be31d065aa0b0d2\": container with ID starting with 86c8300c9da7ec56ef965b37b7ee22ea1ad183861e0552615be31d065aa0b0d2 not found: ID does not exist"
Sep 09 15:59:10.589589 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.589552    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"]
Sep 09 15:59:10.591764 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.591741    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1218fe-6dddf45d95-f2q5z"]
Sep 09 15:59:10.867125 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:10.867048    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64c5e348-bf04-44e1-8427-d01a2e5ad69a" path="/var/lib/kubelet/pods/64c5e348-bf04-44e1-8427-d01a2e5ad69a/volumes"
Sep 09 15:59:11.518788 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:11.518708    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:59:15.531691 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:15.531656    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"]
Sep 09 15:59:15.574210 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:15.574175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:59:30.551901 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:30.551840    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c" podUID="de66609a-367c-4f7b-8d89-8467a6af4012" containerName="main" containerID="cri-o://4e335f37333b241e3a250bb6d6df1eee89f078be7a1274dc96116e1181126d8d" gracePeriod=45
Sep 09 15:59:30.669041 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:30.669009    2578 generic.go:356] "Generic (PLEG): container finished" podID="de66609a-367c-4f7b-8d89-8467a6af4012" containerID="4e335f37333b241e3a250bb6d6df1eee89f078be7a1274dc96116e1181126d8d" exitCode=0
Sep 09 15:59:30.669041 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:30.669055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c" event={"ID":"de66609a-367c-4f7b-8d89-8467a6af4012","Type":"ContainerDied","Data":"4e335f37333b241e3a250bb6d6df1eee89f078be7a1274dc96116e1181126d8d"}
Sep 09 15:59:30.819161 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:30.819103    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:59:30.945411 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:30.945375    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-model-cache\") pod \"de66609a-367c-4f7b-8d89-8467a6af4012\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") "
Sep 09 15:59:30.945637 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:30.945457    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de66609a-367c-4f7b-8d89-8467a6af4012-tls-certs\") pod \"de66609a-367c-4f7b-8d89-8467a6af4012\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") "
Sep 09 15:59:30.945637 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:30.945515    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pq6p\" (UniqueName: \"kubernetes.io/projected/de66609a-367c-4f7b-8d89-8467a6af4012-kube-api-access-4pq6p\") pod \"de66609a-367c-4f7b-8d89-8467a6af4012\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") "
Sep 09 15:59:30.945637 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:30.945547    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-dshm\") pod \"de66609a-367c-4f7b-8d89-8467a6af4012\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") "
Sep 09 15:59:30.945637 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:30.945594    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-home\") pod \"de66609a-367c-4f7b-8d89-8467a6af4012\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") "
Sep 09 15:59:30.945876 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:30.945647    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-tmp-dir\") pod \"de66609a-367c-4f7b-8d89-8467a6af4012\" (UID: \"de66609a-367c-4f7b-8d89-8467a6af4012\") "
Sep 09 15:59:30.945876 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:30.945644    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-model-cache" (OuterVolumeSpecName: "model-cache") pod "de66609a-367c-4f7b-8d89-8467a6af4012" (UID: "de66609a-367c-4f7b-8d89-8467a6af4012"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:59:30.945968 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:30.945908    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-home" (OuterVolumeSpecName: "home") pod "de66609a-367c-4f7b-8d89-8467a6af4012" (UID: "de66609a-367c-4f7b-8d89-8467a6af4012"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:59:30.945968 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:30.945903    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de66609a-367c-4f7b-8d89-8467a6af4012" (UID: "de66609a-367c-4f7b-8d89-8467a6af4012"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:59:30.945968 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:30.945928    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-model-cache\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:59:30.947904 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:30.947879    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-dshm" (OuterVolumeSpecName: "dshm") pod "de66609a-367c-4f7b-8d89-8467a6af4012" (UID: "de66609a-367c-4f7b-8d89-8467a6af4012"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 15:59:30.948045 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:30.947928    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de66609a-367c-4f7b-8d89-8467a6af4012-kube-api-access-4pq6p" (OuterVolumeSpecName: "kube-api-access-4pq6p") pod "de66609a-367c-4f7b-8d89-8467a6af4012" (UID: "de66609a-367c-4f7b-8d89-8467a6af4012"). InnerVolumeSpecName "kube-api-access-4pq6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 15:59:30.948045 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:30.948026    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de66609a-367c-4f7b-8d89-8467a6af4012-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de66609a-367c-4f7b-8d89-8467a6af4012" (UID: "de66609a-367c-4f7b-8d89-8467a6af4012"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 15:59:31.047324 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:31.047289    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de66609a-367c-4f7b-8d89-8467a6af4012-tls-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:59:31.047324 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:31.047317    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pq6p\" (UniqueName: \"kubernetes.io/projected/de66609a-367c-4f7b-8d89-8467a6af4012-kube-api-access-4pq6p\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:59:31.047324 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:31.047326    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-dshm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:59:31.047324 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:31.047335    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-home\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:59:31.047585 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:31.047342    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de66609a-367c-4f7b-8d89-8467a6af4012-tmp-dir\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 15:59:31.675997 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:31.675965    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"
Sep 09 15:59:31.676397 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:31.676011    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c" event={"ID":"de66609a-367c-4f7b-8d89-8467a6af4012","Type":"ContainerDied","Data":"83b1c3cf596759442a5e12d2ee06410babb49698bf7a6e7897e6370d7f1e69b9"}
Sep 09 15:59:31.676397 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:31.676067    2578 scope.go:117] "RemoveContainer" containerID="4e335f37333b241e3a250bb6d6df1eee89f078be7a1274dc96116e1181126d8d"
Sep 09 15:59:31.696207 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:31.696181    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"]
Sep 09 15:59:31.699623 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:31.699601    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-1218fe/ipp-route-1218fe-kserve-c89f4f655-fw94c"]
Sep 09 15:59:32.867035 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:32.867006    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de66609a-367c-4f7b-8d89-8467a6af4012" path="/var/lib/kubelet/pods/de66609a-367c-4f7b-8d89-8467a6af4012/volumes"
Sep 09 15:59:46.378778 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:46.378741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:59:46.434238 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:46.434210    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 15:59:48.951237 ip-10-0-134-183 kubenswrapper[2578]: I0909 15:59:48.951208    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:03.268051 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:03.268014    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:03.714560 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:03.714522    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:04.090907 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:04.090814    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:04.194919 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:04.194866    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:07.226929 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:07.226888    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:11.180038 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:11.180006    2578 scope.go:117] "RemoveContainer" containerID="6eac61ebe20ca698a00e411fbb070dad238a38e6df8bff87d6318e6528405da5"
Sep 09 16:00:13.830921 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:13.830884    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:14.368535 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.368506    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"]
Sep 09 16:00:14.368753 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.368709    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7" podUID="18557905-9aed-41eb-bf3e-8f58d93eb8f2" containerName="maas-api" containerID="cri-o://e141e8db547fc18544e58c8648f8859c6a08e1d3a2dd32700d511ef53cced9c1" gracePeriod=30
Sep 09 16:00:14.401413 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.401384    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:14.484011 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.483967    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:14.612310 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.612290    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 16:00:14.641299 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.641277    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18557905-9aed-41eb-bf3e-8f58d93eb8f2-openshift-service-ca\") pod \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\" (UID: \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\") "
Sep 09 16:00:14.641423 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.641317    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18557905-9aed-41eb-bf3e-8f58d93eb8f2-maas-api-tls\") pod \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\" (UID: \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\") "
Sep 09 16:00:14.641423 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.641347    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18557905-9aed-41eb-bf3e-8f58d93eb8f2-metrics-certs\") pod \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\" (UID: \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\") "
Sep 09 16:00:14.641423 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.641365    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wb5j\" (UniqueName: \"kubernetes.io/projected/18557905-9aed-41eb-bf3e-8f58d93eb8f2-kube-api-access-4wb5j\") pod \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\" (UID: \"18557905-9aed-41eb-bf3e-8f58d93eb8f2\") "
Sep 09 16:00:14.641688 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.641659    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18557905-9aed-41eb-bf3e-8f58d93eb8f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "18557905-9aed-41eb-bf3e-8f58d93eb8f2" (UID: "18557905-9aed-41eb-bf3e-8f58d93eb8f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:00:14.643568 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.643543    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18557905-9aed-41eb-bf3e-8f58d93eb8f2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "18557905-9aed-41eb-bf3e-8f58d93eb8f2" (UID: "18557905-9aed-41eb-bf3e-8f58d93eb8f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:00:14.643672 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.643610    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18557905-9aed-41eb-bf3e-8f58d93eb8f2-kube-api-access-4wb5j" (OuterVolumeSpecName: "kube-api-access-4wb5j") pod "18557905-9aed-41eb-bf3e-8f58d93eb8f2" (UID: "18557905-9aed-41eb-bf3e-8f58d93eb8f2"). InnerVolumeSpecName "kube-api-access-4wb5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:00:14.643672 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.643653    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18557905-9aed-41eb-bf3e-8f58d93eb8f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "18557905-9aed-41eb-bf3e-8f58d93eb8f2" (UID: "18557905-9aed-41eb-bf3e-8f58d93eb8f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:00:14.742284 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.742245    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18557905-9aed-41eb-bf3e-8f58d93eb8f2-openshift-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:00:14.742284 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.742274    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18557905-9aed-41eb-bf3e-8f58d93eb8f2-maas-api-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:00:14.742284 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.742284    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18557905-9aed-41eb-bf3e-8f58d93eb8f2-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:00:14.742284 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.742292    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wb5j\" (UniqueName: \"kubernetes.io/projected/18557905-9aed-41eb-bf3e-8f58d93eb8f2-kube-api-access-4wb5j\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:00:14.893023 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.892925    2578 generic.go:356] "Generic (PLEG): container finished" podID="18557905-9aed-41eb-bf3e-8f58d93eb8f2" containerID="e141e8db547fc18544e58c8648f8859c6a08e1d3a2dd32700d511ef53cced9c1" exitCode=0
Sep 09 16:00:14.893023 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.892962    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7" event={"ID":"18557905-9aed-41eb-bf3e-8f58d93eb8f2","Type":"ContainerDied","Data":"e141e8db547fc18544e58c8648f8859c6a08e1d3a2dd32700d511ef53cced9c1"}
Sep 09 16:00:14.893023 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.892968    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"
Sep 09 16:00:14.893023 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.893024    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7" event={"ID":"18557905-9aed-41eb-bf3e-8f58d93eb8f2","Type":"ContainerDied","Data":"c2c5dc9f52aedbba04389e583739c01a65442406996775fc4ef7c1812b1688f9"}
Sep 09 16:00:14.893552 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.893047    2578 scope.go:117] "RemoveContainer" containerID="e141e8db547fc18544e58c8648f8859c6a08e1d3a2dd32700d511ef53cced9c1"
Sep 09 16:00:14.901796 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.901777    2578 scope.go:117] "RemoveContainer" containerID="e141e8db547fc18544e58c8648f8859c6a08e1d3a2dd32700d511ef53cced9c1"
Sep 09 16:00:14.902104 ip-10-0-134-183 kubenswrapper[2578]: E0909 16:00:14.902081    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e141e8db547fc18544e58c8648f8859c6a08e1d3a2dd32700d511ef53cced9c1\": container with ID starting with e141e8db547fc18544e58c8648f8859c6a08e1d3a2dd32700d511ef53cced9c1 not found: ID does not exist" containerID="e141e8db547fc18544e58c8648f8859c6a08e1d3a2dd32700d511ef53cced9c1"
Sep 09 16:00:14.902174 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.902107    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e141e8db547fc18544e58c8648f8859c6a08e1d3a2dd32700d511ef53cced9c1"} err="failed to get container status \"e141e8db547fc18544e58c8648f8859c6a08e1d3a2dd32700d511ef53cced9c1\": rpc error: code = NotFound desc = could not find container \"e141e8db547fc18544e58c8648f8859c6a08e1d3a2dd32700d511ef53cced9c1\": container with ID starting with e141e8db547fc18544e58c8648f8859c6a08e1d3a2dd32700d511ef53cced9c1 not found: ID does not exist"
Sep 09 16:00:14.909490 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.909467    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"]
Sep 09 16:00:14.915249 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:14.915227    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a2b442-bf4944b86-gw8l7"]
Sep 09 16:00:15.280734 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:15.280663    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 16:00:15.286344 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:15.286322    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 16:00:15.287946 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:15.287924    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 16:00:15.293347 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:15.293330    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 16:00:15.810715 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:15.810658    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:16.868088 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:16.868059    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18557905-9aed-41eb-bf3e-8f58d93eb8f2" path="/var/lib/kubelet/pods/18557905-9aed-41eb-bf3e-8f58d93eb8f2/volumes"
Sep 09 16:00:19.289547 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:19.289515    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:20.881541 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:20.881501    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:20.942820 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:20.942790    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:21.165842 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:21.165795    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:21.676606 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:21.676577    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:31.975105 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:31.974898    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"]
Sep 09 16:00:31.975856 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:31.975800    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p" podUID="6b36a70e-acc8-4a14-89ff-262b0c9048ad" containerName="maas-api" containerID="cri-o://e6371b6da2f24c22a255e2c54a5c32acf35e3869f87705c38ebfb547a99dfc4d" gracePeriod=30
Sep 09 16:00:32.007218 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.007175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:32.109529 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.109492    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:32.232753 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.232694    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 16:00:32.311372 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.311336    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8g6b\" (UniqueName: \"kubernetes.io/projected/6b36a70e-acc8-4a14-89ff-262b0c9048ad-kube-api-access-n8g6b\") pod \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\" (UID: \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\") "
Sep 09 16:00:32.311527 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.311397    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b36a70e-acc8-4a14-89ff-262b0c9048ad-metrics-certs\") pod \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\" (UID: \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\") "
Sep 09 16:00:32.311527 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.311446    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b36a70e-acc8-4a14-89ff-262b0c9048ad-maas-api-tls\") pod \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\" (UID: \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\") "
Sep 09 16:00:32.311527 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.311491    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b36a70e-acc8-4a14-89ff-262b0c9048ad-openshift-service-ca\") pod \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\" (UID: \"6b36a70e-acc8-4a14-89ff-262b0c9048ad\") "
Sep 09 16:00:32.311882 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.311860    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b36a70e-acc8-4a14-89ff-262b0c9048ad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6b36a70e-acc8-4a14-89ff-262b0c9048ad" (UID: "6b36a70e-acc8-4a14-89ff-262b0c9048ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:00:32.313573 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.313540    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b36a70e-acc8-4a14-89ff-262b0c9048ad-kube-api-access-n8g6b" (OuterVolumeSpecName: "kube-api-access-n8g6b") pod "6b36a70e-acc8-4a14-89ff-262b0c9048ad" (UID: "6b36a70e-acc8-4a14-89ff-262b0c9048ad"). InnerVolumeSpecName "kube-api-access-n8g6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:00:32.313687 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.313615    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b36a70e-acc8-4a14-89ff-262b0c9048ad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6b36a70e-acc8-4a14-89ff-262b0c9048ad" (UID: "6b36a70e-acc8-4a14-89ff-262b0c9048ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:00:32.313687 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.313649    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b36a70e-acc8-4a14-89ff-262b0c9048ad-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6b36a70e-acc8-4a14-89ff-262b0c9048ad" (UID: "6b36a70e-acc8-4a14-89ff-262b0c9048ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:00:32.413150 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.413117    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8g6b\" (UniqueName: \"kubernetes.io/projected/6b36a70e-acc8-4a14-89ff-262b0c9048ad-kube-api-access-n8g6b\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:00:32.413150 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.413145    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b36a70e-acc8-4a14-89ff-262b0c9048ad-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:00:32.413150 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.413154    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b36a70e-acc8-4a14-89ff-262b0c9048ad-maas-api-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:00:32.413377 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.413162    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b36a70e-acc8-4a14-89ff-262b0c9048ad-openshift-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:00:32.984282 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.984246    2578 generic.go:356] "Generic (PLEG): container finished" podID="6b36a70e-acc8-4a14-89ff-262b0c9048ad" containerID="e6371b6da2f24c22a255e2c54a5c32acf35e3869f87705c38ebfb547a99dfc4d" exitCode=0
Sep 09 16:00:32.984676 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.984291    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"
Sep 09 16:00:32.984676 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.984327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p" event={"ID":"6b36a70e-acc8-4a14-89ff-262b0c9048ad","Type":"ContainerDied","Data":"e6371b6da2f24c22a255e2c54a5c32acf35e3869f87705c38ebfb547a99dfc4d"}
Sep 09 16:00:32.984676 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.984380    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p" event={"ID":"6b36a70e-acc8-4a14-89ff-262b0c9048ad","Type":"ContainerDied","Data":"3ba80407b8c8482481d87985260d26201ee561d0b42e097939369320c25a1ceb"}
Sep 09 16:00:32.984676 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.984402    2578 scope.go:117] "RemoveContainer" containerID="e6371b6da2f24c22a255e2c54a5c32acf35e3869f87705c38ebfb547a99dfc4d"
Sep 09 16:00:32.993852 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.993829    2578 scope.go:117] "RemoveContainer" containerID="e6371b6da2f24c22a255e2c54a5c32acf35e3869f87705c38ebfb547a99dfc4d"
Sep 09 16:00:32.994124 ip-10-0-134-183 kubenswrapper[2578]: E0909 16:00:32.994105    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6371b6da2f24c22a255e2c54a5c32acf35e3869f87705c38ebfb547a99dfc4d\": container with ID starting with e6371b6da2f24c22a255e2c54a5c32acf35e3869f87705c38ebfb547a99dfc4d not found: ID does not exist" containerID="e6371b6da2f24c22a255e2c54a5c32acf35e3869f87705c38ebfb547a99dfc4d"
Sep 09 16:00:32.994184 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:32.994134    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6371b6da2f24c22a255e2c54a5c32acf35e3869f87705c38ebfb547a99dfc4d"} err="failed to get container status \"e6371b6da2f24c22a255e2c54a5c32acf35e3869f87705c38ebfb547a99dfc4d\": rpc error: code = NotFound desc = could not find container \"e6371b6da2f24c22a255e2c54a5c32acf35e3869f87705c38ebfb547a99dfc4d\": container with ID starting with e6371b6da2f24c22a255e2c54a5c32acf35e3869f87705c38ebfb547a99dfc4d not found: ID does not exist"
Sep 09 16:00:33.000541 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:33.000518    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"]
Sep 09 16:00:33.004326 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:33.004304    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-450a51-644b6bbb57-z868p"]
Sep 09 16:00:33.977524 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:33.977490    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:34.044625 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:34.044592    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:34.867219 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:34.867186    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b36a70e-acc8-4a14-89ff-262b0c9048ad" path="/var/lib/kubelet/pods/6b36a70e-acc8-4a14-89ff-262b0c9048ad/volumes"
Sep 09 16:00:38.407350 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:38.407316    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:38.476157 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:38.476123    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:38.685959 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:38.685884    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:40.662612 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:40.662583    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:49.640747 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:49.640710    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:49.717672 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:49.717640    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:49.817743 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:49.817712    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:49.918270 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:49.918178    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:50.241854 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:50.241774    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:56.004437 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:56.004408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:56.105061 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:56.105033    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:56.287638 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:56.287558    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:00:59.161731 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:59.161692    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f6f9bc65f-sw57k"]
Sep 09 16:00:59.162248 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:59.161944    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7f6f9bc65f-sw57k" podUID="93263f27-3bc8-4744-b5d2-91fd622adf08" containerName="manager" containerID="cri-o://715eaaa9f5458a6a29113a7f8197d72ab3ca4b95477a4c93a9cd25bbee8e571e" gracePeriod=30
Sep 09 16:00:59.408074 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:59.408041    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f9bc65f-sw57k"
Sep 09 16:00:59.471530 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:59.471456    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93263f27-3bc8-4744-b5d2-91fd622adf08-metrics-certs\") pod \"93263f27-3bc8-4744-b5d2-91fd622adf08\" (UID: \"93263f27-3bc8-4744-b5d2-91fd622adf08\") "
Sep 09 16:00:59.471530 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:59.471496    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93263f27-3bc8-4744-b5d2-91fd622adf08-webhook-cert\") pod \"93263f27-3bc8-4744-b5d2-91fd622adf08\" (UID: \"93263f27-3bc8-4744-b5d2-91fd622adf08\") "
Sep 09 16:00:59.471725 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:59.471536    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nmkk\" (UniqueName: \"kubernetes.io/projected/93263f27-3bc8-4744-b5d2-91fd622adf08-kube-api-access-4nmkk\") pod \"93263f27-3bc8-4744-b5d2-91fd622adf08\" (UID: \"93263f27-3bc8-4744-b5d2-91fd622adf08\") "
Sep 09 16:00:59.473704 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:59.473677    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93263f27-3bc8-4744-b5d2-91fd622adf08-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "93263f27-3bc8-4744-b5d2-91fd622adf08" (UID: "93263f27-3bc8-4744-b5d2-91fd622adf08"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:00:59.473854 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:59.473698    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93263f27-3bc8-4744-b5d2-91fd622adf08-kube-api-access-4nmkk" (OuterVolumeSpecName: "kube-api-access-4nmkk") pod "93263f27-3bc8-4744-b5d2-91fd622adf08" (UID: "93263f27-3bc8-4744-b5d2-91fd622adf08"). InnerVolumeSpecName "kube-api-access-4nmkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:00:59.473854 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:59.473754    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93263f27-3bc8-4744-b5d2-91fd622adf08-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "93263f27-3bc8-4744-b5d2-91fd622adf08" (UID: "93263f27-3bc8-4744-b5d2-91fd622adf08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:00:59.572364 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:59.572326    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93263f27-3bc8-4744-b5d2-91fd622adf08-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:00:59.572364 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:59.572359    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93263f27-3bc8-4744-b5d2-91fd622adf08-webhook-cert\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:00:59.572364 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:00:59.572370    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nmkk\" (UniqueName: \"kubernetes.io/projected/93263f27-3bc8-4744-b5d2-91fd622adf08-kube-api-access-4nmkk\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:01:00.127937 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:01:00.127904    2578 generic.go:356] "Generic (PLEG): container finished" podID="93263f27-3bc8-4744-b5d2-91fd622adf08" containerID="715eaaa9f5458a6a29113a7f8197d72ab3ca4b95477a4c93a9cd25bbee8e571e" exitCode=0
Sep 09 16:01:00.128171 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:01:00.127951    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f9bc65f-sw57k"
Sep 09 16:01:00.128171 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:01:00.128017    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f9bc65f-sw57k" event={"ID":"93263f27-3bc8-4744-b5d2-91fd622adf08","Type":"ContainerDied","Data":"715eaaa9f5458a6a29113a7f8197d72ab3ca4b95477a4c93a9cd25bbee8e571e"}
Sep 09 16:01:00.128171 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:01:00.128073    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f9bc65f-sw57k" event={"ID":"93263f27-3bc8-4744-b5d2-91fd622adf08","Type":"ContainerDied","Data":"e2f90b8358484b1eb4efcb7d4149e44cf507ef1ef3e0fe082bdce0e69771dbed"}
Sep 09 16:01:00.128171 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:01:00.128104    2578 scope.go:117] "RemoveContainer" containerID="715eaaa9f5458a6a29113a7f8197d72ab3ca4b95477a4c93a9cd25bbee8e571e"
Sep 09 16:01:00.141404 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:01:00.141386    2578 scope.go:117] "RemoveContainer" containerID="715eaaa9f5458a6a29113a7f8197d72ab3ca4b95477a4c93a9cd25bbee8e571e"
Sep 09 16:01:00.141649 ip-10-0-134-183 kubenswrapper[2578]: E0909 16:01:00.141633    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"715eaaa9f5458a6a29113a7f8197d72ab3ca4b95477a4c93a9cd25bbee8e571e\": container with ID starting with 715eaaa9f5458a6a29113a7f8197d72ab3ca4b95477a4c93a9cd25bbee8e571e not found: ID does not exist" containerID="715eaaa9f5458a6a29113a7f8197d72ab3ca4b95477a4c93a9cd25bbee8e571e"
Sep 09 16:01:00.141710 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:01:00.141658    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"715eaaa9f5458a6a29113a7f8197d72ab3ca4b95477a4c93a9cd25bbee8e571e"} err="failed to get container status \"715eaaa9f5458a6a29113a7f8197d72ab3ca4b95477a4c93a9cd25bbee8e571e\": rpc error: code = NotFound desc = could not find container \"715eaaa9f5458a6a29113a7f8197d72ab3ca4b95477a4c93a9cd25bbee8e571e\": container with ID starting with 715eaaa9f5458a6a29113a7f8197d72ab3ca4b95477a4c93a9cd25bbee8e571e not found: ID does not exist"
Sep 09 16:01:00.148585 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:01:00.148561    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f6f9bc65f-sw57k"]
Sep 09 16:01:00.152321 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:01:00.152302    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f6f9bc65f-sw57k"]
Sep 09 16:01:00.868420 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:01:00.868389    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93263f27-3bc8-4744-b5d2-91fd622adf08" path="/var/lib/kubelet/pods/93263f27-3bc8-4744-b5d2-91fd622adf08/volumes"
Sep 09 16:02:00.977382 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977350    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f6f9bc65f-b9n9q"]
Sep 09 16:02:00.977795 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977764    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de66609a-367c-4f7b-8d89-8467a6af4012" containerName="main"
Sep 09 16:02:00.977795 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977776    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="de66609a-367c-4f7b-8d89-8467a6af4012" containerName="main"
Sep 09 16:02:00.977795 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977791    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b36a70e-acc8-4a14-89ff-262b0c9048ad" containerName="maas-api"
Sep 09 16:02:00.977795 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977796    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b36a70e-acc8-4a14-89ff-262b0c9048ad" containerName="maas-api"
Sep 09 16:02:00.977926 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977806    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b752b7c7-1dc5-47b4-ae90-4a619c4038b4" containerName="maas-api"
Sep 09 16:02:00.977926 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977814    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b752b7c7-1dc5-47b4-ae90-4a619c4038b4" containerName="maas-api"
Sep 09 16:02:00.977926 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977825    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93263f27-3bc8-4744-b5d2-91fd622adf08" containerName="manager"
Sep 09 16:02:00.977926 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977834    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="93263f27-3bc8-4744-b5d2-91fd622adf08" containerName="manager"
Sep 09 16:02:00.977926 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977848    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18557905-9aed-41eb-bf3e-8f58d93eb8f2" containerName="maas-api"
Sep 09 16:02:00.977926 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977854    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="18557905-9aed-41eb-bf3e-8f58d93eb8f2" containerName="maas-api"
Sep 09 16:02:00.977926 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977862    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64c5e348-bf04-44e1-8427-d01a2e5ad69a" containerName="maas-api"
Sep 09 16:02:00.977926 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977868    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="64c5e348-bf04-44e1-8427-d01a2e5ad69a" containerName="maas-api"
Sep 09 16:02:00.978187 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977933    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b36a70e-acc8-4a14-89ff-262b0c9048ad" containerName="maas-api"
Sep 09 16:02:00.978187 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977942    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="64c5e348-bf04-44e1-8427-d01a2e5ad69a" containerName="maas-api"
Sep 09 16:02:00.978187 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977947    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b752b7c7-1dc5-47b4-ae90-4a619c4038b4" containerName="maas-api"
Sep 09 16:02:00.978187 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977954    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="93263f27-3bc8-4744-b5d2-91fd622adf08" containerName="manager"
Sep 09 16:02:00.978187 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977960    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="18557905-9aed-41eb-bf3e-8f58d93eb8f2" containerName="maas-api"
Sep 09 16:02:00.978187 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.977965    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="de66609a-367c-4f7b-8d89-8467a6af4012" containerName="main"
Sep 09 16:02:00.982061 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.982045    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q"
Sep 09 16:02:00.985430 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.985406    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 16:02:00.985577 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.985421    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 16:02:00.985577 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.985430    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bsjmm\""
Sep 09 16:02:00.985577 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.985444    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 16:02:00.991496 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:00.991473    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f6f9bc65f-b9n9q"]
Sep 09 16:02:01.140080 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:01.140044    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33a74b51-45f5-46e3-ae35-c39dfc43bcd1-metrics-certs\") pod \"maas-controller-7f6f9bc65f-b9n9q\" (UID: \"33a74b51-45f5-46e3-ae35-c39dfc43bcd1\") " pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q"
Sep 09 16:02:01.140080 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:01.140082    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/33a74b51-45f5-46e3-ae35-c39dfc43bcd1-webhook-cert\") pod \"maas-controller-7f6f9bc65f-b9n9q\" (UID: \"33a74b51-45f5-46e3-ae35-c39dfc43bcd1\") " pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q"
Sep 09 16:02:01.140295 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:01.140108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj8sh\" (UniqueName: \"kubernetes.io/projected/33a74b51-45f5-46e3-ae35-c39dfc43bcd1-kube-api-access-xj8sh\") pod \"maas-controller-7f6f9bc65f-b9n9q\" (UID: \"33a74b51-45f5-46e3-ae35-c39dfc43bcd1\") " pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q"
Sep 09 16:02:01.240878 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:01.240790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33a74b51-45f5-46e3-ae35-c39dfc43bcd1-metrics-certs\") pod \"maas-controller-7f6f9bc65f-b9n9q\" (UID: \"33a74b51-45f5-46e3-ae35-c39dfc43bcd1\") " pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q"
Sep 09 16:02:01.240878 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:01.240826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/33a74b51-45f5-46e3-ae35-c39dfc43bcd1-webhook-cert\") pod \"maas-controller-7f6f9bc65f-b9n9q\" (UID: \"33a74b51-45f5-46e3-ae35-c39dfc43bcd1\") " pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q"
Sep 09 16:02:01.240878 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:01.240842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj8sh\" (UniqueName: \"kubernetes.io/projected/33a74b51-45f5-46e3-ae35-c39dfc43bcd1-kube-api-access-xj8sh\") pod \"maas-controller-7f6f9bc65f-b9n9q\" (UID: \"33a74b51-45f5-46e3-ae35-c39dfc43bcd1\") " pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q"
Sep 09 16:02:01.243526 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:01.243498    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33a74b51-45f5-46e3-ae35-c39dfc43bcd1-metrics-certs\") pod \"maas-controller-7f6f9bc65f-b9n9q\" (UID: \"33a74b51-45f5-46e3-ae35-c39dfc43bcd1\") " pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q"
Sep 09 16:02:01.243526 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:01.243522    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/33a74b51-45f5-46e3-ae35-c39dfc43bcd1-webhook-cert\") pod \"maas-controller-7f6f9bc65f-b9n9q\" (UID: \"33a74b51-45f5-46e3-ae35-c39dfc43bcd1\") " pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q"
Sep 09 16:02:01.248532 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:01.248507    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj8sh\" (UniqueName: \"kubernetes.io/projected/33a74b51-45f5-46e3-ae35-c39dfc43bcd1-kube-api-access-xj8sh\") pod \"maas-controller-7f6f9bc65f-b9n9q\" (UID: \"33a74b51-45f5-46e3-ae35-c39dfc43bcd1\") " pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q"
Sep 09 16:02:01.294467 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:01.294433    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q"
Sep 09 16:02:01.571584 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:01.571556    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f6f9bc65f-b9n9q"]
Sep 09 16:02:01.573107 ip-10-0-134-183 kubenswrapper[2578]: W0909 16:02:01.573081    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a74b51_45f5_46e3_ae35_c39dfc43bcd1.slice/crio-7e12fb8f3c637ae3085f6ac82bc9abe49633b8fc5c16b65a81cbe3e907f947be WatchSource:0}: Error finding container 7e12fb8f3c637ae3085f6ac82bc9abe49633b8fc5c16b65a81cbe3e907f947be: Status 404 returned error can't find the container with id 7e12fb8f3c637ae3085f6ac82bc9abe49633b8fc5c16b65a81cbe3e907f947be
Sep 09 16:02:01.574841 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:01.574821    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:02:02.428884 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:02.428846    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q" event={"ID":"33a74b51-45f5-46e3-ae35-c39dfc43bcd1","Type":"ContainerStarted","Data":"7e12fb8f3c637ae3085f6ac82bc9abe49633b8fc5c16b65a81cbe3e907f947be"}
Sep 09 16:02:06.449711 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:06.449670    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q" event={"ID":"33a74b51-45f5-46e3-ae35-c39dfc43bcd1","Type":"ContainerStarted","Data":"e8f6bdc533927b0d4b73141298796e9831bfdf30061cd0b4375dde9c92d98e61"}
Sep 09 16:02:06.450182 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:06.449784    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q"
Sep 09 16:02:06.466089 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:06.466041    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q" podStartSLOduration=2.196682374 podStartE2EDuration="6.466025861s" totalImagesPullingTime="4.269343487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:02:06.464686223 +0000 UTC m=+2516.106728212" watchObservedRunningTime="2026-09-09 16:02:06.466025861 +0000 UTC m=+2516.108067863"
Sep 09 16:02:11.220441 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:11.220409    2578 scope.go:117] "RemoveContainer" containerID="09a52fe85271b5cc01f759ac2f3cfe68af2e245b75c7838d37dd90fa61d1a880"
Sep 09 16:02:12.393585 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:12.393551    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:02:12.444464 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:12.444429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:02:17.456963 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:17.456934    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q"
Sep 09 16:02:24.271377 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:24.271329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:02:24.595972 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:24.595886    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:02:24.717618 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:24.717569    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f6f9bc65f-b9n9q"]
Sep 09 16:02:24.717903 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:24.717856    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q" podUID="33a74b51-45f5-46e3-ae35-c39dfc43bcd1" containerName="manager" containerID="cri-o://e8f6bdc533927b0d4b73141298796e9831bfdf30061cd0b4375dde9c92d98e61" gracePeriod=30
Sep 09 16:02:24.965896 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:24.965873    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q"
Sep 09 16:02:25.060727 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:25.060700    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj8sh\" (UniqueName: \"kubernetes.io/projected/33a74b51-45f5-46e3-ae35-c39dfc43bcd1-kube-api-access-xj8sh\") pod \"33a74b51-45f5-46e3-ae35-c39dfc43bcd1\" (UID: \"33a74b51-45f5-46e3-ae35-c39dfc43bcd1\") "
Sep 09 16:02:25.060893 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:25.060808    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/33a74b51-45f5-46e3-ae35-c39dfc43bcd1-webhook-cert\") pod \"33a74b51-45f5-46e3-ae35-c39dfc43bcd1\" (UID: \"33a74b51-45f5-46e3-ae35-c39dfc43bcd1\") "
Sep 09 16:02:25.060893 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:25.060842    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33a74b51-45f5-46e3-ae35-c39dfc43bcd1-metrics-certs\") pod \"33a74b51-45f5-46e3-ae35-c39dfc43bcd1\" (UID: \"33a74b51-45f5-46e3-ae35-c39dfc43bcd1\") "
Sep 09 16:02:25.062939 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:25.062908    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a74b51-45f5-46e3-ae35-c39dfc43bcd1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "33a74b51-45f5-46e3-ae35-c39dfc43bcd1" (UID: "33a74b51-45f5-46e3-ae35-c39dfc43bcd1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:02:25.063080 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:25.063006    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a74b51-45f5-46e3-ae35-c39dfc43bcd1-kube-api-access-xj8sh" (OuterVolumeSpecName: "kube-api-access-xj8sh") pod "33a74b51-45f5-46e3-ae35-c39dfc43bcd1" (UID: "33a74b51-45f5-46e3-ae35-c39dfc43bcd1"). InnerVolumeSpecName "kube-api-access-xj8sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:02:25.063080 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:25.063025    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a74b51-45f5-46e3-ae35-c39dfc43bcd1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "33a74b51-45f5-46e3-ae35-c39dfc43bcd1" (UID: "33a74b51-45f5-46e3-ae35-c39dfc43bcd1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:02:25.161484 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:25.161459    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33a74b51-45f5-46e3-ae35-c39dfc43bcd1-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:02:25.161484 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:25.161485    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj8sh\" (UniqueName: \"kubernetes.io/projected/33a74b51-45f5-46e3-ae35-c39dfc43bcd1-kube-api-access-xj8sh\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:02:25.161669 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:25.161495    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/33a74b51-45f5-46e3-ae35-c39dfc43bcd1-webhook-cert\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:02:25.548678 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:25.548593    2578 generic.go:356] "Generic (PLEG): container finished" podID="33a74b51-45f5-46e3-ae35-c39dfc43bcd1" containerID="e8f6bdc533927b0d4b73141298796e9831bfdf30061cd0b4375dde9c92d98e61" exitCode=0
Sep 09 16:02:25.548678 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:25.548635    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q"
Sep 09 16:02:25.549174 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:25.548679    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q" event={"ID":"33a74b51-45f5-46e3-ae35-c39dfc43bcd1","Type":"ContainerDied","Data":"e8f6bdc533927b0d4b73141298796e9831bfdf30061cd0b4375dde9c92d98e61"}
Sep 09 16:02:25.549174 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:25.548730    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f9bc65f-b9n9q" event={"ID":"33a74b51-45f5-46e3-ae35-c39dfc43bcd1","Type":"ContainerDied","Data":"7e12fb8f3c637ae3085f6ac82bc9abe49633b8fc5c16b65a81cbe3e907f947be"}
Sep 09 16:02:25.549174 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:25.548754    2578 scope.go:117] "RemoveContainer" containerID="e8f6bdc533927b0d4b73141298796e9831bfdf30061cd0b4375dde9c92d98e61"
Sep 09 16:02:25.560670 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:25.560654    2578 scope.go:117] "RemoveContainer" containerID="e8f6bdc533927b0d4b73141298796e9831bfdf30061cd0b4375dde9c92d98e61"
Sep 09 16:02:25.560947 ip-10-0-134-183 kubenswrapper[2578]: E0909 16:02:25.560924    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f6bdc533927b0d4b73141298796e9831bfdf30061cd0b4375dde9c92d98e61\": container with ID starting with e8f6bdc533927b0d4b73141298796e9831bfdf30061cd0b4375dde9c92d98e61 not found: ID does not exist" containerID="e8f6bdc533927b0d4b73141298796e9831bfdf30061cd0b4375dde9c92d98e61"
Sep 09 16:02:25.561049 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:25.560960    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f6bdc533927b0d4b73141298796e9831bfdf30061cd0b4375dde9c92d98e61"} err="failed to get container status \"e8f6bdc533927b0d4b73141298796e9831bfdf30061cd0b4375dde9c92d98e61\": rpc error: code = NotFound desc = could not find container \"e8f6bdc533927b0d4b73141298796e9831bfdf30061cd0b4375dde9c92d98e61\": container with ID starting with e8f6bdc533927b0d4b73141298796e9831bfdf30061cd0b4375dde9c92d98e61 not found: ID does not exist"
Sep 09 16:02:25.569380 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:25.569357    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f6f9bc65f-b9n9q"]
Sep 09 16:02:25.575568 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:25.575547    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f6f9bc65f-b9n9q"]
Sep 09 16:02:26.408910 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.408879    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:02:26.537114 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.537075    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f6f9bc65f-gkllf"]
Sep 09 16:02:26.537577 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.537558    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33a74b51-45f5-46e3-ae35-c39dfc43bcd1" containerName="manager"
Sep 09 16:02:26.537671 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.537579    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="33a74b51-45f5-46e3-ae35-c39dfc43bcd1" containerName="manager"
Sep 09 16:02:26.537729 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.537669    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="33a74b51-45f5-46e3-ae35-c39dfc43bcd1" containerName="manager"
Sep 09 16:02:26.542236 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.542215    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f9bc65f-gkllf"
Sep 09 16:02:26.544739 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.544717    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 16:02:26.544858 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.544839    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 16:02:26.545619 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.545601    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 16:02:26.545715 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.545605    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bsjmm\""
Sep 09 16:02:26.550381 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.550358    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f6f9bc65f-gkllf"]
Sep 09 16:02:26.677159 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.677070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp79c\" (UniqueName: \"kubernetes.io/projected/ae3d7c64-d05c-41d7-866a-d5f3119efb96-kube-api-access-cp79c\") pod \"maas-controller-7f6f9bc65f-gkllf\" (UID: \"ae3d7c64-d05c-41d7-866a-d5f3119efb96\") " pod="opendatahub/maas-controller-7f6f9bc65f-gkllf"
Sep 09 16:02:26.677159 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.677110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae3d7c64-d05c-41d7-866a-d5f3119efb96-metrics-certs\") pod \"maas-controller-7f6f9bc65f-gkllf\" (UID: \"ae3d7c64-d05c-41d7-866a-d5f3119efb96\") " pod="opendatahub/maas-controller-7f6f9bc65f-gkllf"
Sep 09 16:02:26.677349 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.677207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae3d7c64-d05c-41d7-866a-d5f3119efb96-webhook-cert\") pod \"maas-controller-7f6f9bc65f-gkllf\" (UID: \"ae3d7c64-d05c-41d7-866a-d5f3119efb96\") " pod="opendatahub/maas-controller-7f6f9bc65f-gkllf"
Sep 09 16:02:26.778579 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.778532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae3d7c64-d05c-41d7-866a-d5f3119efb96-webhook-cert\") pod \"maas-controller-7f6f9bc65f-gkllf\" (UID: \"ae3d7c64-d05c-41d7-866a-d5f3119efb96\") " pod="opendatahub/maas-controller-7f6f9bc65f-gkllf"
Sep 09 16:02:26.778770 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.778618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp79c\" (UniqueName: \"kubernetes.io/projected/ae3d7c64-d05c-41d7-866a-d5f3119efb96-kube-api-access-cp79c\") pod \"maas-controller-7f6f9bc65f-gkllf\" (UID: \"ae3d7c64-d05c-41d7-866a-d5f3119efb96\") " pod="opendatahub/maas-controller-7f6f9bc65f-gkllf"
Sep 09 16:02:26.778770 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.778641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae3d7c64-d05c-41d7-866a-d5f3119efb96-metrics-certs\") pod \"maas-controller-7f6f9bc65f-gkllf\" (UID: \"ae3d7c64-d05c-41d7-866a-d5f3119efb96\") " pod="opendatahub/maas-controller-7f6f9bc65f-gkllf"
Sep 09 16:02:26.781267 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.781242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae3d7c64-d05c-41d7-866a-d5f3119efb96-webhook-cert\") pod \"maas-controller-7f6f9bc65f-gkllf\" (UID: \"ae3d7c64-d05c-41d7-866a-d5f3119efb96\") " pod="opendatahub/maas-controller-7f6f9bc65f-gkllf"
Sep 09 16:02:26.781378 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.781299    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae3d7c64-d05c-41d7-866a-d5f3119efb96-metrics-certs\") pod \"maas-controller-7f6f9bc65f-gkllf\" (UID: \"ae3d7c64-d05c-41d7-866a-d5f3119efb96\") " pod="opendatahub/maas-controller-7f6f9bc65f-gkllf"
Sep 09 16:02:26.786409 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.786384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp79c\" (UniqueName: \"kubernetes.io/projected/ae3d7c64-d05c-41d7-866a-d5f3119efb96-kube-api-access-cp79c\") pod \"maas-controller-7f6f9bc65f-gkllf\" (UID: \"ae3d7c64-d05c-41d7-866a-d5f3119efb96\") " pod="opendatahub/maas-controller-7f6f9bc65f-gkllf"
Sep 09 16:02:26.854894 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.854848    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f9bc65f-gkllf"
Sep 09 16:02:26.868637 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:26.868611    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33a74b51-45f5-46e3-ae35-c39dfc43bcd1" path="/var/lib/kubelet/pods/33a74b51-45f5-46e3-ae35-c39dfc43bcd1/volumes"
Sep 09 16:02:27.126185 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:27.126111    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f6f9bc65f-gkllf"]
Sep 09 16:02:27.129097 ip-10-0-134-183 kubenswrapper[2578]: W0909 16:02:27.129070    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3d7c64_d05c_41d7_866a_d5f3119efb96.slice/crio-a546b8d90ee88bcb1ab37c737f5cdc5dc7a22c89bedd02ce25d2e864cfcc3e6f WatchSource:0}: Error finding container a546b8d90ee88bcb1ab37c737f5cdc5dc7a22c89bedd02ce25d2e864cfcc3e6f: Status 404 returned error can't find the container with id a546b8d90ee88bcb1ab37c737f5cdc5dc7a22c89bedd02ce25d2e864cfcc3e6f
Sep 09 16:02:27.563566 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:27.563542    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f9bc65f-gkllf" event={"ID":"ae3d7c64-d05c-41d7-866a-d5f3119efb96","Type":"ContainerStarted","Data":"a546b8d90ee88bcb1ab37c737f5cdc5dc7a22c89bedd02ce25d2e864cfcc3e6f"}
Sep 09 16:02:28.569378 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:28.569346    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f9bc65f-gkllf" event={"ID":"ae3d7c64-d05c-41d7-866a-d5f3119efb96","Type":"ContainerStarted","Data":"17cb4281303ea1e82445f330ed197b5a728ce70a95fb28e2c0b1153fa378bb5e"}
Sep 09 16:02:28.569746 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:28.569443    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f6f9bc65f-gkllf"
Sep 09 16:02:28.585253 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:28.585207    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f6f9bc65f-gkllf" podStartSLOduration=2.201881316 podStartE2EDuration="2.58519383s" totalImagesPullingTime="383.312514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:02:28.58402823 +0000 UTC m=+2538.226070234" watchObservedRunningTime="2026-09-09 16:02:28.58519383 +0000 UTC m=+2538.227235819"
Sep 09 16:02:39.576951 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:02:39.576920    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f6f9bc65f-gkllf"
Sep 09 16:03:58.635845 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:03:58.635809    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:19.901372 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:04:19.901336    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:58.488526 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:04:58.488473    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:04:58.550663 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:04:58.550635    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:06.452356 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.452275    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"]
Sep 09 16:05:06.455954 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.455932    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.458267 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.458247    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a46aea87-kserve-self-signed-certs\""
Sep 09 16:05:06.462938 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.462914    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"]
Sep 09 16:05:06.480934 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.480909    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:06.539739 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.539700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-dshm\") pod \"e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") " pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.539903 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.539767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-tmp-dir\") pod \"e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") " pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.539903 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.539830    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-tls-certs\") pod \"e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") " pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.539903 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.539897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-home\") pod \"e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") " pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.540071 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.539912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-model-cache\") pod \"e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") " pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.540071 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.539928    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x4z4\" (UniqueName: \"kubernetes.io/projected/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-kube-api-access-7x4z4\") pod \"e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") " pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.641272 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.641233    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-home\") pod \"e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") " pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.641272 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.641270    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-model-cache\") pod \"e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") " pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.641528 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.641288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x4z4\" (UniqueName: \"kubernetes.io/projected/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-kube-api-access-7x4z4\") pod \"e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") " pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.641528 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.641322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-dshm\") pod \"e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") " pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.641528 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.641515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-tmp-dir\") pod \"e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") " pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.641710 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.641571    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-tls-certs\") pod \"e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") " pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.641778 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.641739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-model-cache\") pod \"e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") " pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.641839 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.641799    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-home\") pod \"e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") " pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.641972 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.641913    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-tmp-dir\") pod \"e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") " pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.643833 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.643809    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-dshm\") pod \"e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") " pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.644217 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.644199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-tls-certs\") pod \"e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") " pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.648658 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.648632    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x4z4\" (UniqueName: \"kubernetes.io/projected/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-kube-api-access-7x4z4\") pod \"e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") " pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:06.780044 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:06.779945    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:07.048573 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:07.048552    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"]
Sep 09 16:05:07.050874 ip-10-0-134-183 kubenswrapper[2578]: W0909 16:05:07.050845    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc4a4917_7574_4cc6_88ac_a00360bfcc4e.slice/crio-fbc5d0491b904bc0f125cf2e4601d965bf28b9cfc7561eb0b595c5197e8036f7 WatchSource:0}: Error finding container fbc5d0491b904bc0f125cf2e4601d965bf28b9cfc7561eb0b595c5197e8036f7: Status 404 returned error can't find the container with id fbc5d0491b904bc0f125cf2e4601d965bf28b9cfc7561eb0b595c5197e8036f7
Sep 09 16:05:07.340479 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:07.340391    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96" event={"ID":"dc4a4917-7574-4cc6-88ac-a00360bfcc4e","Type":"ContainerStarted","Data":"4be3296a207d6f24dfdb42d42e78356793f30d0084ce7468381222eca880439e"}
Sep 09 16:05:07.340479 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:07.340433    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96" event={"ID":"dc4a4917-7574-4cc6-88ac-a00360bfcc4e","Type":"ContainerStarted","Data":"fbc5d0491b904bc0f125cf2e4601d965bf28b9cfc7561eb0b595c5197e8036f7"}
Sep 09 16:05:07.356562 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:07.356513    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96" podStartSLOduration=1.356496584 podStartE2EDuration="1.356496584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:05:07.354971841 +0000 UTC m=+2696.997013835" watchObservedRunningTime="2026-09-09 16:05:07.356496584 +0000 UTC m=+2696.998538573"
Sep 09 16:05:15.313905 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:15.313877    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 16:05:15.319697 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:15.319674    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 16:05:15.321528 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:15.321509    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 16:05:15.326215 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:15.326198    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42r95_8e8d5b7a-7aa0-40dd-951e-76e34bf395cb/ovn-acl-logging/0.log"
Sep 09 16:05:16.780379 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:16.780340    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:16.780802 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:16.780584    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:16.793199 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:16.793180    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:17.400621 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:17.400584    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:19.286510 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:19.286476    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:39.921328 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:39.921291    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"]
Sep 09 16:05:39.941296 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:39.941250    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:41.536791 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:41.536761    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:05:54.943920 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:54.943859    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96" podUID="dc4a4917-7574-4cc6-88ac-a00360bfcc4e" containerName="main" containerID="cri-o://4be3296a207d6f24dfdb42d42e78356793f30d0084ce7468381222eca880439e" gracePeriod=45
Sep 09 16:05:55.190388 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.190364    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:55.303391 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.303306    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-home\") pod \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") "
Sep 09 16:05:55.303533 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.303402    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-tls-certs\") pod \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") "
Sep 09 16:05:55.303533 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.303447    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x4z4\" (UniqueName: \"kubernetes.io/projected/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-kube-api-access-7x4z4\") pod \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") "
Sep 09 16:05:55.303533 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.303507    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-model-cache\") pod \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") "
Sep 09 16:05:55.303702 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.303551    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-dshm\") pod \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") "
Sep 09 16:05:55.303702 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.303581    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-tmp-dir\") pod \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\" (UID: \"dc4a4917-7574-4cc6-88ac-a00360bfcc4e\") "
Sep 09 16:05:55.303702 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.303584    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-home" (OuterVolumeSpecName: "home") pod "dc4a4917-7574-4cc6-88ac-a00360bfcc4e" (UID: "dc4a4917-7574-4cc6-88ac-a00360bfcc4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:05:55.303849 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.303766    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc4a4917-7574-4cc6-88ac-a00360bfcc4e" (UID: "dc4a4917-7574-4cc6-88ac-a00360bfcc4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:05:55.303902 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.303889    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc4a4917-7574-4cc6-88ac-a00360bfcc4e" (UID: "dc4a4917-7574-4cc6-88ac-a00360bfcc4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:05:55.303944 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.303912    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-home\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:05:55.303944 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.303928    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-model-cache\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:05:55.305913 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.305880    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-dshm" (OuterVolumeSpecName: "dshm") pod "dc4a4917-7574-4cc6-88ac-a00360bfcc4e" (UID: "dc4a4917-7574-4cc6-88ac-a00360bfcc4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:05:55.305913 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.305903    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc4a4917-7574-4cc6-88ac-a00360bfcc4e" (UID: "dc4a4917-7574-4cc6-88ac-a00360bfcc4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:05:55.306099 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.306021    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-kube-api-access-7x4z4" (OuterVolumeSpecName: "kube-api-access-7x4z4") pod "dc4a4917-7574-4cc6-88ac-a00360bfcc4e" (UID: "dc4a4917-7574-4cc6-88ac-a00360bfcc4e"). InnerVolumeSpecName "kube-api-access-7x4z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:05:55.404621 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.404583    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-tls-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:05:55.404621 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.404617    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x4z4\" (UniqueName: \"kubernetes.io/projected/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-kube-api-access-7x4z4\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:05:55.404811 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.404630    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-dshm\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:05:55.404811 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.404645    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc4a4917-7574-4cc6-88ac-a00360bfcc4e-tmp-dir\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:05:55.578178 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.578092    2578 generic.go:356] "Generic (PLEG): container finished" podID="dc4a4917-7574-4cc6-88ac-a00360bfcc4e" containerID="4be3296a207d6f24dfdb42d42e78356793f30d0084ce7468381222eca880439e" exitCode=0
Sep 09 16:05:55.578178 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.578137    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"
Sep 09 16:05:55.578178 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.578160    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96" event={"ID":"dc4a4917-7574-4cc6-88ac-a00360bfcc4e","Type":"ContainerDied","Data":"4be3296a207d6f24dfdb42d42e78356793f30d0084ce7468381222eca880439e"}
Sep 09 16:05:55.578449 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.578189    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96" event={"ID":"dc4a4917-7574-4cc6-88ac-a00360bfcc4e","Type":"ContainerDied","Data":"fbc5d0491b904bc0f125cf2e4601d965bf28b9cfc7561eb0b595c5197e8036f7"}
Sep 09 16:05:55.578449 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.578206    2578 scope.go:117] "RemoveContainer" containerID="4be3296a207d6f24dfdb42d42e78356793f30d0084ce7468381222eca880439e"
Sep 09 16:05:55.587597 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.587580    2578 scope.go:117] "RemoveContainer" containerID="4be3296a207d6f24dfdb42d42e78356793f30d0084ce7468381222eca880439e"
Sep 09 16:05:55.587895 ip-10-0-134-183 kubenswrapper[2578]: E0909 16:05:55.587875    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be3296a207d6f24dfdb42d42e78356793f30d0084ce7468381222eca880439e\": container with ID starting with 4be3296a207d6f24dfdb42d42e78356793f30d0084ce7468381222eca880439e not found: ID does not exist" containerID="4be3296a207d6f24dfdb42d42e78356793f30d0084ce7468381222eca880439e"
Sep 09 16:05:55.587945 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.587903    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be3296a207d6f24dfdb42d42e78356793f30d0084ce7468381222eca880439e"} err="failed to get container status \"4be3296a207d6f24dfdb42d42e78356793f30d0084ce7468381222eca880439e\": rpc error: code = NotFound desc = could not find container \"4be3296a207d6f24dfdb42d42e78356793f30d0084ce7468381222eca880439e\": container with ID starting with 4be3296a207d6f24dfdb42d42e78356793f30d0084ce7468381222eca880439e not found: ID does not exist"
Sep 09 16:05:55.598440 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.598419    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"]
Sep 09 16:05:55.601437 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:55.601416    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a46aea87-kserve-6d88d495c5-q2p96"]
Sep 09 16:05:56.867566 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:05:56.867532    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc4a4917-7574-4cc6-88ac-a00360bfcc4e" path="/var/lib/kubelet/pods/dc4a4917-7574-4cc6-88ac-a00360bfcc4e/volumes"
Sep 09 16:06:03.227952 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.227920    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"]
Sep 09 16:06:03.228450 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.228332    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc4a4917-7574-4cc6-88ac-a00360bfcc4e" containerName="main"
Sep 09 16:06:03.228450 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.228345    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc4a4917-7574-4cc6-88ac-a00360bfcc4e" containerName="main"
Sep 09 16:06:03.228450 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.228409    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc4a4917-7574-4cc6-88ac-a00360bfcc4e" containerName="main"
Sep 09 16:06:03.232140 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.232120    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"
Sep 09 16:06:03.245583 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.245555    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"]
Sep 09 16:06:03.278925 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.278891    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae16573-3f23-4842-ac4b-db27abf88972-maas-api-tls\") pod \"maas-api-8476d9f544-qwtl5\" (UID: \"1ae16573-3f23-4842-ac4b-db27abf88972\") " pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"
Sep 09 16:06:03.278925 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.278922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae16573-3f23-4842-ac4b-db27abf88972-openshift-service-ca\") pod \"maas-api-8476d9f544-qwtl5\" (UID: \"1ae16573-3f23-4842-ac4b-db27abf88972\") " pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"
Sep 09 16:06:03.279141 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.279037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7wwj\" (UniqueName: \"kubernetes.io/projected/1ae16573-3f23-4842-ac4b-db27abf88972-kube-api-access-q7wwj\") pod \"maas-api-8476d9f544-qwtl5\" (UID: \"1ae16573-3f23-4842-ac4b-db27abf88972\") " pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"
Sep 09 16:06:03.279141 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.279115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae16573-3f23-4842-ac4b-db27abf88972-metrics-certs\") pod \"maas-api-8476d9f544-qwtl5\" (UID: \"1ae16573-3f23-4842-ac4b-db27abf88972\") " pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"
Sep 09 16:06:03.379681 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.379644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7wwj\" (UniqueName: \"kubernetes.io/projected/1ae16573-3f23-4842-ac4b-db27abf88972-kube-api-access-q7wwj\") pod \"maas-api-8476d9f544-qwtl5\" (UID: \"1ae16573-3f23-4842-ac4b-db27abf88972\") " pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"
Sep 09 16:06:03.379861 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.379719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae16573-3f23-4842-ac4b-db27abf88972-metrics-certs\") pod \"maas-api-8476d9f544-qwtl5\" (UID: \"1ae16573-3f23-4842-ac4b-db27abf88972\") " pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"
Sep 09 16:06:03.379861 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.379783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae16573-3f23-4842-ac4b-db27abf88972-maas-api-tls\") pod \"maas-api-8476d9f544-qwtl5\" (UID: \"1ae16573-3f23-4842-ac4b-db27abf88972\") " pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"
Sep 09 16:06:03.379861 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.379802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae16573-3f23-4842-ac4b-db27abf88972-openshift-service-ca\") pod \"maas-api-8476d9f544-qwtl5\" (UID: \"1ae16573-3f23-4842-ac4b-db27abf88972\") " pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"
Sep 09 16:06:03.380507 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.380484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae16573-3f23-4842-ac4b-db27abf88972-openshift-service-ca\") pod \"maas-api-8476d9f544-qwtl5\" (UID: \"1ae16573-3f23-4842-ac4b-db27abf88972\") " pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"
Sep 09 16:06:03.382419 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.382396    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae16573-3f23-4842-ac4b-db27abf88972-maas-api-tls\") pod \"maas-api-8476d9f544-qwtl5\" (UID: \"1ae16573-3f23-4842-ac4b-db27abf88972\") " pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"
Sep 09 16:06:03.382540 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.382523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae16573-3f23-4842-ac4b-db27abf88972-metrics-certs\") pod \"maas-api-8476d9f544-qwtl5\" (UID: \"1ae16573-3f23-4842-ac4b-db27abf88972\") " pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"
Sep 09 16:06:03.388363 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.388337    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7wwj\" (UniqueName: \"kubernetes.io/projected/1ae16573-3f23-4842-ac4b-db27abf88972-kube-api-access-q7wwj\") pod \"maas-api-8476d9f544-qwtl5\" (UID: \"1ae16573-3f23-4842-ac4b-db27abf88972\") " pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"
Sep 09 16:06:03.544585 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.544508    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"
Sep 09 16:06:03.824665 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:03.824634    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"]
Sep 09 16:06:03.827100 ip-10-0-134-183 kubenswrapper[2578]: W0909 16:06:03.827067    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae16573_3f23_4842_ac4b_db27abf88972.slice/crio-d9a0b0479e9f3663e12bff659db9eee094c2b0999eb7eada9fce897d07d345ef WatchSource:0}: Error finding container d9a0b0479e9f3663e12bff659db9eee094c2b0999eb7eada9fce897d07d345ef: Status 404 returned error can't find the container with id d9a0b0479e9f3663e12bff659db9eee094c2b0999eb7eada9fce897d07d345ef
Sep 09 16:06:04.627930 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:04.627883    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5" event={"ID":"1ae16573-3f23-4842-ac4b-db27abf88972","Type":"ContainerStarted","Data":"7797485194b739504603c3348a733a739f135f054683c967b1569f3e4d0cd6d3"}
Sep 09 16:06:04.628320 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:04.627951    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5" event={"ID":"1ae16573-3f23-4842-ac4b-db27abf88972","Type":"ContainerStarted","Data":"d9a0b0479e9f3663e12bff659db9eee094c2b0999eb7eada9fce897d07d345ef"}
Sep 09 16:06:04.628320 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:04.627972    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"
Sep 09 16:06:04.643893 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:04.643837    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5" podStartSLOduration=0.992593339 podStartE2EDuration="1.643820805s" totalImagesPullingTime="651.227466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:06:04.643273609 +0000 UTC m=+2754.285315618" watchObservedRunningTime="2026-09-09 16:06:04.643820805 +0000 UTC m=+2754.285862793"
Sep 09 16:06:10.639257 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:10.639221    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8476d9f544-qwtl5"
Sep 09 16:06:19.319447 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:19.319409    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"]
Sep 09 16:06:19.319965 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:19.319626    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf" podUID="0f6ab1fe-334f-42dd-8b08-fe3b91701a10" containerName="maas-api" containerID="cri-o://319537013a699f14925adae4c159c1363a8ec64f5683309bfd8d55038f44ddb5" gracePeriod=30
Sep 09 16:06:20.465095 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.465033    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf" podUID="0f6ab1fe-334f-42dd-8b08-fe3b91701a10" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.40:8443/health\": dial tcp 10.132.0.40:8443: connect: connection refused"
Sep 09 16:06:20.572879 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.572855    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"
Sep 09 16:06:20.707910 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.707878    2578 generic.go:356] "Generic (PLEG): container finished" podID="0f6ab1fe-334f-42dd-8b08-fe3b91701a10" containerID="319537013a699f14925adae4c159c1363a8ec64f5683309bfd8d55038f44ddb5" exitCode=0
Sep 09 16:06:20.708120 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.707925    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"
Sep 09 16:06:20.708120 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.707936    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf" event={"ID":"0f6ab1fe-334f-42dd-8b08-fe3b91701a10","Type":"ContainerDied","Data":"319537013a699f14925adae4c159c1363a8ec64f5683309bfd8d55038f44ddb5"}
Sep 09 16:06:20.708120 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.707990    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d864d7888-wltcf" event={"ID":"0f6ab1fe-334f-42dd-8b08-fe3b91701a10","Type":"ContainerDied","Data":"04fc6baaaa603178fbea6481ad1c6aa7a5a69858a0800c03ddf5b2f4333d7edf"}
Sep 09 16:06:20.708120 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.708004    2578 scope.go:117] "RemoveContainer" containerID="319537013a699f14925adae4c159c1363a8ec64f5683309bfd8d55038f44ddb5"
Sep 09 16:06:20.716789 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.716767    2578 scope.go:117] "RemoveContainer" containerID="319537013a699f14925adae4c159c1363a8ec64f5683309bfd8d55038f44ddb5"
Sep 09 16:06:20.717089 ip-10-0-134-183 kubenswrapper[2578]: E0909 16:06:20.717070    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319537013a699f14925adae4c159c1363a8ec64f5683309bfd8d55038f44ddb5\": container with ID starting with 319537013a699f14925adae4c159c1363a8ec64f5683309bfd8d55038f44ddb5 not found: ID does not exist" containerID="319537013a699f14925adae4c159c1363a8ec64f5683309bfd8d55038f44ddb5"
Sep 09 16:06:20.717137 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.717104    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319537013a699f14925adae4c159c1363a8ec64f5683309bfd8d55038f44ddb5"} err="failed to get container status \"319537013a699f14925adae4c159c1363a8ec64f5683309bfd8d55038f44ddb5\": rpc error: code = NotFound desc = could not find container \"319537013a699f14925adae4c159c1363a8ec64f5683309bfd8d55038f44ddb5\": container with ID starting with 319537013a699f14925adae4c159c1363a8ec64f5683309bfd8d55038f44ddb5 not found: ID does not exist"
Sep 09 16:06:20.746550 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.746517    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-maas-api-tls\") pod \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\" (UID: \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\") "
Sep 09 16:06:20.746704 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.746575    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jphwb\" (UniqueName: \"kubernetes.io/projected/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-kube-api-access-jphwb\") pod \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\" (UID: \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\") "
Sep 09 16:06:20.746704 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.746611    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-openshift-service-ca\") pod \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\" (UID: \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\") "
Sep 09 16:06:20.746704 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.746633    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-metrics-certs\") pod \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\" (UID: \"0f6ab1fe-334f-42dd-8b08-fe3b91701a10\") "
Sep 09 16:06:20.746968 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.746941    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f6ab1fe-334f-42dd-8b08-fe3b91701a10" (UID: "0f6ab1fe-334f-42dd-8b08-fe3b91701a10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:06:20.748815 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.748792    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-kube-api-access-jphwb" (OuterVolumeSpecName: "kube-api-access-jphwb") pod "0f6ab1fe-334f-42dd-8b08-fe3b91701a10" (UID: "0f6ab1fe-334f-42dd-8b08-fe3b91701a10"). InnerVolumeSpecName "kube-api-access-jphwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:06:20.748892 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.748820    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f6ab1fe-334f-42dd-8b08-fe3b91701a10" (UID: "0f6ab1fe-334f-42dd-8b08-fe3b91701a10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:06:20.748931 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.748887    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0f6ab1fe-334f-42dd-8b08-fe3b91701a10" (UID: "0f6ab1fe-334f-42dd-8b08-fe3b91701a10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:06:20.848352 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.848318    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-metrics-certs\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:06:20.848352 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.848347    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-maas-api-tls\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:06:20.848352 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.848356    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jphwb\" (UniqueName: \"kubernetes.io/projected/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-kube-api-access-jphwb\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:06:20.848668 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:20.848367    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6ab1fe-334f-42dd-8b08-fe3b91701a10-openshift-service-ca\") on node \"ip-10-0-134-183.ec2.internal\" DevicePath \"\""
Sep 09 16:06:21.027153 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:21.027058    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"]
Sep 09 16:06:21.030645 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:21.030615    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d864d7888-wltcf"]
Sep 09 16:06:22.867174 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:22.867139    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f6ab1fe-334f-42dd-8b08-fe3b91701a10" path="/var/lib/kubelet/pods/0f6ab1fe-334f-42dd-8b08-fe3b91701a10/volumes"
Sep 09 16:06:25.725176 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:25.725137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:06:39.210523 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:39.210437    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f6f9bc65f-gkllf_ae3d7c64-d05c-41d7-866a-d5f3119efb96/manager/0.log"
Sep 09 16:06:39.849373 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:39.849344    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74fc54996-lvfk9_24e1e39c-65d8-4473-8696-78b2fd866d1d/copy-manifests/0.log"
Sep 09 16:06:39.856245 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:39.856215    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74fc54996-lvfk9_24e1e39c-65d8-4473-8696-78b2fd866d1d/manager/0.log"
Sep 09 16:06:40.198127 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:40.198100    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8476d9f544-qwtl5_1ae16573-3f23-4842-ac4b-db27abf88972/maas-api/0.log"
Sep 09 16:06:40.660412 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:40.660360    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-qcp97_b94dd558-d97c-430b-a20b-494b4d89e3fa/postgres/0.log"
Sep 09 16:06:41.716214 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:41.716183    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-snwd7_88fb0e01-a115-4c3b-b495-fe57bddfd9e0/manager/0.log"
Sep 09 16:06:42.934352 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:42.934320    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59598c6787-t5n82_22dcf369-0bc5-4588-a4ae-c2b38cfc879f/kube-auth-proxy/0.log"
Sep 09 16:06:43.052367 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:43.052337    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-c784m_d41c51d4-c973-4e22-a0a1-96c523e020f3/istio-proxy/1.log"
Sep 09 16:06:43.295260 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:43.295162    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-qf4s4_c743a297-9127-4f66-8153-8a00d9327040/payload-processing/0.log"
Sep 09 16:06:44.105940 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:44.105904    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-2zsxj_d3d4615a-378a-4737-9f51-723397da6166/main/0.log"
Sep 09 16:06:44.331308 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:44.331251    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-pjsm5_0e8d5797-6341-4a87-b2dd-81185f930a2e/main/0.log"
Sep 09 16:06:50.584260 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:50.584216    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:06:50.684728 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:50.684693    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:06:53.651068 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:53.651032    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h4r6t_560288f5-fbb4-4d6b-84bd-0987afe4c482/global-pull-secret-syncer/0.log"
Sep 09 16:06:53.780407 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:53.780377    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log"
Sep 09 16:06:53.884424 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:53.884397    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-183.ec2.internal_f5569ed1edc68219f13a87c1afb087c8/haproxy/0.log"
Sep 09 16:06:54.125604 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:54.125516    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7kt4h_588589aa-effd-4129-96af-6180f4a1b5d8/konnectivity-agent/0.log" limit=1024
Sep 09 16:06:58.128609 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:06:58.128529    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-snwd7_88fb0e01-a115-4c3b-b495-fe57bddfd9e0/manager/0.log"
Sep 09 16:07:00.089026 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:00.088937    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d053f8b6-bff5-4174-acd4-8ea1a06f9192/alertmanager/0.log"
Sep 09 16:07:00.110331 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:00.110293    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d053f8b6-bff5-4174-acd4-8ea1a06f9192/config-reloader/0.log"
Sep 09 16:07:00.133126 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:00.133097    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d053f8b6-bff5-4174-acd4-8ea1a06f9192/kube-rbac-proxy-web/0.log"
Sep 09 16:07:00.161804 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:00.161766    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d053f8b6-bff5-4174-acd4-8ea1a06f9192/kube-rbac-proxy/0.log"
Sep 09 16:07:00.188757 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:00.188726    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d053f8b6-bff5-4174-acd4-8ea1a06f9192/kube-rbac-proxy-metric/0.log"
Sep 09 16:07:00.213367 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:00.213335    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d053f8b6-bff5-4174-acd4-8ea1a06f9192/prom-label-proxy/0.log"
Sep 09 16:07:00.235843 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:00.235815    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d053f8b6-bff5-4174-acd4-8ea1a06f9192/init-config-reloader/0.log"
Sep 09 16:07:00.276523 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:00.276494    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-ntzdf_3f4e6b8e-8887-4c02-bc23-4bdc6227850a/cluster-monitoring-operator/0.log"
Sep 09 16:07:00.413302 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:00.413271    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-2l7x5_2070b616-f62d-4ba0-a0ab-eb4bd6a81cbc/monitoring-plugin/0.log"
Sep 09 16:07:00.443054 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:00.443027    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-66kk7_e455fe37-13c1-47ff-bc2c-71730947cea9/node-exporter/0.log"
Sep 09 16:07:00.467923 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:00.467897    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-66kk7_e455fe37-13c1-47ff-bc2c-71730947cea9/kube-rbac-proxy/0.log"
Sep 09 16:07:00.492854 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:00.492828    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-66kk7_e455fe37-13c1-47ff-bc2c-71730947cea9/init-textfile/0.log"
Sep 09 16:07:00.886729 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:00.886687    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d5b37ecf-f4b1-478c-93c9-7157e7a5a140/prometheus/0.log"
Sep 09 16:07:00.907148 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:00.907121    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d5b37ecf-f4b1-478c-93c9-7157e7a5a140/config-reloader/0.log"
Sep 09 16:07:00.929503 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:00.929428    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d5b37ecf-f4b1-478c-93c9-7157e7a5a140/thanos-sidecar/0.log"
Sep 09 16:07:00.950456 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:00.950427    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d5b37ecf-f4b1-478c-93c9-7157e7a5a140/kube-rbac-proxy-web/0.log"
Sep 09 16:07:00.973715 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:00.973687    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d5b37ecf-f4b1-478c-93c9-7157e7a5a140/kube-rbac-proxy/0.log"
Sep 09 16:07:00.997061 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:00.997025    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d5b37ecf-f4b1-478c-93c9-7157e7a5a140/kube-rbac-proxy-thanos/0.log"
Sep 09 16:07:01.019533 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:01.019509    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d5b37ecf-f4b1-478c-93c9-7157e7a5a140/init-config-reloader/0.log"
Sep 09 16:07:01.050152 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:01.050116    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-txjcq_ed66c965-aa5c-41e1-b2ba-6e9abe252472/prometheus-operator/0.log"
Sep 09 16:07:01.072740 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:01.072692    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-txjcq_ed66c965-aa5c-41e1-b2ba-6e9abe252472/kube-rbac-proxy/0.log"
Sep 09 16:07:01.097891 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:01.097866    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-4zzgj_e6e7e9e2-4a62-47f9-a603-1c77e465e43a/prometheus-operator-admission-webhook/0.log"
Sep 09 16:07:01.129213 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:01.129186    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-797d4dcc75-8z4fk_1981bf38-60c0-4674-bb1e-cf6fc10eabff/telemeter-client/0.log"
Sep 09 16:07:01.155524 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:01.155499    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-797d4dcc75-8z4fk_1981bf38-60c0-4674-bb1e-cf6fc10eabff/reload/0.log"
Sep 09 16:07:01.179006 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:01.178961    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-797d4dcc75-8z4fk_1981bf38-60c0-4674-bb1e-cf6fc10eabff/kube-rbac-proxy/0.log"
Sep 09 16:07:03.123995 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:03.123958    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-62m26_18973d3c-7fa4-4b71-ade6-1cd976413494/networking-console-plugin/0.log"
Sep 09 16:07:03.761677 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:03.761645    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/2.log"
Sep 09 16:07:03.766024 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:03.765969    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vsgld_dc2e93bc-cc28-454c-9526-13e2837cdd14/console-operator/3.log"
Sep 09 16:07:04.299851 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.299819    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-ll79v_22748ed4-a6ec-4cad-a2a6-f3a10c1d52a4/download-server/0.log"
Sep 09 16:07:04.522279 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.522244    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"]
Sep 09 16:07:04.522854 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.522768    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f6ab1fe-334f-42dd-8b08-fe3b91701a10" containerName="maas-api"
Sep 09 16:07:04.522854 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.522793    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6ab1fe-334f-42dd-8b08-fe3b91701a10" containerName="maas-api"
Sep 09 16:07:04.523102 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.522897    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f6ab1fe-334f-42dd-8b08-fe3b91701a10" containerName="maas-api"
Sep 09 16:07:04.526338 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.526315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:04.528674 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.528649    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kmsmj\"/\"openshift-service-ca.crt\""
Sep 09 16:07:04.528808 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.528707    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kmsmj\"/\"default-dockercfg-2c896\""
Sep 09 16:07:04.529570 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.529548    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kmsmj\"/\"kube-root-ca.crt\""
Sep 09 16:07:04.532416 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.532394    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"]
Sep 09 16:07:04.660939 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.660905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7338a1b5-6676-4295-bb07-93bdb93ead02-sys\") pod \"perf-node-gather-daemonset-k7pbf\" (UID: \"7338a1b5-6676-4295-bb07-93bdb93ead02\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:04.660939 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.660939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7338a1b5-6676-4295-bb07-93bdb93ead02-proc\") pod \"perf-node-gather-daemonset-k7pbf\" (UID: \"7338a1b5-6676-4295-bb07-93bdb93ead02\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:04.661180 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.661041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7338a1b5-6676-4295-bb07-93bdb93ead02-lib-modules\") pod \"perf-node-gather-daemonset-k7pbf\" (UID: \"7338a1b5-6676-4295-bb07-93bdb93ead02\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:04.661180 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.661168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gdf2\" (UniqueName: \"kubernetes.io/projected/7338a1b5-6676-4295-bb07-93bdb93ead02-kube-api-access-9gdf2\") pod \"perf-node-gather-daemonset-k7pbf\" (UID: \"7338a1b5-6676-4295-bb07-93bdb93ead02\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:04.661258 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.661198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7338a1b5-6676-4295-bb07-93bdb93ead02-podres\") pod \"perf-node-gather-daemonset-k7pbf\" (UID: \"7338a1b5-6676-4295-bb07-93bdb93ead02\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:04.761800 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.761691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7338a1b5-6676-4295-bb07-93bdb93ead02-sys\") pod \"perf-node-gather-daemonset-k7pbf\" (UID: \"7338a1b5-6676-4295-bb07-93bdb93ead02\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:04.761974 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.761811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7338a1b5-6676-4295-bb07-93bdb93ead02-proc\") pod \"perf-node-gather-daemonset-k7pbf\" (UID: \"7338a1b5-6676-4295-bb07-93bdb93ead02\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:04.761974 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.761778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7338a1b5-6676-4295-bb07-93bdb93ead02-sys\") pod \"perf-node-gather-daemonset-k7pbf\" (UID: \"7338a1b5-6676-4295-bb07-93bdb93ead02\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:04.761974 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.761863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7338a1b5-6676-4295-bb07-93bdb93ead02-lib-modules\") pod \"perf-node-gather-daemonset-k7pbf\" (UID: \"7338a1b5-6676-4295-bb07-93bdb93ead02\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:04.761974 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.761936    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7338a1b5-6676-4295-bb07-93bdb93ead02-proc\") pod \"perf-node-gather-daemonset-k7pbf\" (UID: \"7338a1b5-6676-4295-bb07-93bdb93ead02\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:04.762149 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.762013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gdf2\" (UniqueName: \"kubernetes.io/projected/7338a1b5-6676-4295-bb07-93bdb93ead02-kube-api-access-9gdf2\") pod \"perf-node-gather-daemonset-k7pbf\" (UID: \"7338a1b5-6676-4295-bb07-93bdb93ead02\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:04.762149 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.762022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7338a1b5-6676-4295-bb07-93bdb93ead02-lib-modules\") pod \"perf-node-gather-daemonset-k7pbf\" (UID: \"7338a1b5-6676-4295-bb07-93bdb93ead02\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:04.762149 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.762048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7338a1b5-6676-4295-bb07-93bdb93ead02-podres\") pod \"perf-node-gather-daemonset-k7pbf\" (UID: \"7338a1b5-6676-4295-bb07-93bdb93ead02\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:04.762251 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.762158    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7338a1b5-6676-4295-bb07-93bdb93ead02-podres\") pod \"perf-node-gather-daemonset-k7pbf\" (UID: \"7338a1b5-6676-4295-bb07-93bdb93ead02\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:04.769854 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.769825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gdf2\" (UniqueName: \"kubernetes.io/projected/7338a1b5-6676-4295-bb07-93bdb93ead02-kube-api-access-9gdf2\") pod \"perf-node-gather-daemonset-k7pbf\" (UID: \"7338a1b5-6676-4295-bb07-93bdb93ead02\") " pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:04.837368 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:04.837329    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:05.107704 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:05.107600    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"]
Sep 09 16:07:05.933290 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:05.933260    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf" event={"ID":"7338a1b5-6676-4295-bb07-93bdb93ead02","Type":"ContainerStarted","Data":"e56195e36b36ef9e37c670b37f480c277f658fcb253c37daf911d2b40ab7b434"}
Sep 09 16:07:05.933681 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:05.933306    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf" event={"ID":"7338a1b5-6676-4295-bb07-93bdb93ead02","Type":"ContainerStarted","Data":"15b6a7a71ad242a05ea2ff01f4ccc1e38b8bacf6684f973e28d6169d3f366530"}
Sep 09 16:07:05.933681 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:05.933396    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:05.950687 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:05.950631    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf" podStartSLOduration=1.950613364 podStartE2EDuration="1.950613364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:07:05.949928992 +0000 UTC m=+2815.591971027" watchObservedRunningTime="2026-09-09 16:07:05.950613364 +0000 UTC m=+2815.592655352"
Sep 09 16:07:06.337781 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:06.337700    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rdv4c_ed80dd24-f513-4215-9f41-c33847a78c80/dns/0.log"
Sep 09 16:07:06.358716 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:06.358684    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rdv4c_ed80dd24-f513-4215-9f41-c33847a78c80/kube-rbac-proxy/0.log"
Sep 09 16:07:06.449892 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:06.449855    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v2hbm_9fc01e4b-4a6f-47f0-8220-ca62e74d7250/dns-node-resolver/0.log"
Sep 09 16:07:07.042763 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:07.042727    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qn7pt_0b0bf69a-0e36-446f-8748-a84c778fe36e/node-ca/0.log"
Sep 09 16:07:08.356569 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:08.356534    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59598c6787-t5n82_22dcf369-0bc5-4588-a4ae-c2b38cfc879f/kube-auth-proxy/0.log"
Sep 09 16:07:08.387814 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:08.387778    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-c784m_d41c51d4-c973-4e22-a0a1-96c523e020f3/istio-proxy/0.log"
Sep 09 16:07:08.390558 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:08.390535    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-c784m_d41c51d4-c973-4e22-a0a1-96c523e020f3/istio-proxy/1.log"
Sep 09 16:07:08.455612 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:08.455583    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-qf4s4_c743a297-9127-4f66-8153-8a00d9327040/payload-processing/0.log"
Sep 09 16:07:09.395814 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:09.395782    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j58rw_7b5c31a0-d5d8-4e50-ad6b-e0697ec8c073/serve-healthcheck-canary/0.log"
Sep 09 16:07:10.008203 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:10.008174    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xnkml_85da71af-045e-45d6-adc6-dca483128c4f/kube-rbac-proxy/0.log"
Sep 09 16:07:10.028870 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:10.028841    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xnkml_85da71af-045e-45d6-adc6-dca483128c4f/exporter/0.log"
Sep 09 16:07:10.050006 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:10.049958    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xnkml_85da71af-045e-45d6-adc6-dca483128c4f/extractor/0.log"
Sep 09 16:07:11.948960 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:11.948926    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kmsmj/perf-node-gather-daemonset-k7pbf"
Sep 09 16:07:12.611762 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:12.611727    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f6f9bc65f-gkllf_ae3d7c64-d05c-41d7-866a-d5f3119efb96/manager/0.log"
Sep 09 16:07:12.910719 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:12.910694    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74fc54996-lvfk9_24e1e39c-65d8-4473-8696-78b2fd866d1d/manager/0.log"
Sep 09 16:07:12.931458 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:12.931428    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74fc54996-lvfk9_24e1e39c-65d8-4473-8696-78b2fd866d1d/copy-manifests/0.log"
Sep 09 16:07:14.374270 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:14.374197    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-76f449f6db-l6jdj_5deb874a-0c47-45b8-880d-deedf83d1af2/manager/0.log"
Sep 09 16:07:15.135018 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:15.134998    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59598c6787-t5n82_22dcf369-0bc5-4588-a4ae-c2b38cfc879f/kube-auth-proxy/0.log"
Sep 09 16:07:15.154858 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:15.154822    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-c784m_d41c51d4-c973-4e22-a0a1-96c523e020f3/istio-proxy/0.log"
Sep 09 16:07:15.157234 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:15.157212    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-c784m_d41c51d4-c973-4e22-a0a1-96c523e020f3/istio-proxy/1.log"
Sep 09 16:07:15.204220 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:15.204192    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-qf4s4_c743a297-9127-4f66-8153-8a00d9327040/payload-processing/0.log"
Sep 09 16:07:17.159728 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:17.159693    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59598c6787-t5n82_22dcf369-0bc5-4588-a4ae-c2b38cfc879f/kube-auth-proxy/0.log"
Sep 09 16:07:17.179823 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:17.179800    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-c784m_d41c51d4-c973-4e22-a0a1-96c523e020f3/istio-proxy/0.log"
Sep 09 16:07:17.182346 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:17.182327    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-c784m_d41c51d4-c973-4e22-a0a1-96c523e020f3/istio-proxy/1.log"
Sep 09 16:07:17.231083 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:17.231047    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-qf4s4_c743a297-9127-4f66-8153-8a00d9327040/payload-processing/0.log"
Sep 09 16:07:19.659453 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:19.659415    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-r6c44_5c4d7ec2-3aa1-4a1b-ba22-83028a88db06/migrator/0.log"
Sep 09 16:07:19.679913 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:19.679879    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-r6c44_5c4d7ec2-3aa1-4a1b-ba22-83028a88db06/graceful-termination/0.log"
Sep 09 16:07:21.009792 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:21.009766    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2wkxv_879098a0-7a7e-4b71-b0ee-b7c759162e26/kube-multus-additional-cni-plugins/0.log"
Sep 09 16:07:21.030739 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:21.030713    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2wkxv_879098a0-7a7e-4b71-b0ee-b7c759162e26/egress-router-binary-copy/0.log"
Sep 09 16:07:21.051608 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:21.051585    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2wkxv_879098a0-7a7e-4b71-b0ee-b7c759162e26/cni-plugins/0.log"
Sep 09 16:07:21.070562 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:21.070541    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2wkxv_879098a0-7a7e-4b71-b0ee-b7c759162e26/bond-cni-plugin/0.log"
Sep 09 16:07:21.091501 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:21.091478    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2wkxv_879098a0-7a7e-4b71-b0ee-b7c759162e26/routeoverride-cni/0.log"
Sep 09 16:07:21.110740 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:21.110715    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2wkxv_879098a0-7a7e-4b71-b0ee-b7c759162e26/whereabouts-cni-bincopy/0.log"
Sep 09 16:07:21.132077 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:21.132058    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2wkxv_879098a0-7a7e-4b71-b0ee-b7c759162e26/whereabouts-cni/0.log"
Sep 09 16:07:21.517766 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:21.517735    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fvk5x_bd6109fe-7e99-4d98-87d0-8a6fc95d6938/kube-multus/0.log"
Sep 09 16:07:21.582955 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:21.582919    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nhtcj_7052f7a0-16d5-4115-82ae-34f07e9db9f9/network-metrics-daemon/0.log"
Sep 09 16:07:21.601034 ip-10-0-134-183 kubenswrapper[2578]: I0909 16:07:21.601003    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nhtcj_7052f7a0-16d5-4115-82ae-34f07e9db9f9/kube-rbac-proxy/0.log"