Sep 17 10:51:23.632345 ip-10-0-139-204 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 10:51:23.632355 ip-10-0-139-204 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 10:51:23.632362 ip-10-0-139-204 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 10:51:23.632612 ip-10-0-139-204 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 10:51:33.881449 ip-10-0-139-204 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 10:51:33.881469 ip-10-0-139-204 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c9d1391a9b2e470ebcdce920f1c8be00 --
Sep 17 10:53:25.806163 ip-10-0-139-204 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 10:53:26.209575 ip-10-0-139-204 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 10:53:26.209575 ip-10-0-139-204 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 10:53:26.209575 ip-10-0-139-204 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 10:53:26.209575 ip-10-0-139-204 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 10:53:26.209575 ip-10-0-139-204 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 10:53:26.210481 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.210392    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 10:53:26.214535 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214518    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 10:53:26.214535 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214534    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 10:53:26.214535 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214538    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214542    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214545    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214547    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214551    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214554    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214557    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214559    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214562    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214565    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214567    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214570    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214573    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214575    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214578    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214586    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214589    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214591    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214594    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214596    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 10:53:26.214638 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214599    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214604    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214608    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214611    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214614    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214616    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214619    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214622    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214626    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214630    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214634    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214637    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214640    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214644    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214649    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214652    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214655    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214658    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214661    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 10:53:26.215112 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214664    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214667    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214670    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214673    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214675    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214678    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214681    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214685    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214687    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214690    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214692    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214695    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214698    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214701    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214703    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214706    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214709    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214711    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214714    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214718    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 10:53:26.215684 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214721    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214723    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214726    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214728    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214731    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214734    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214736    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214739    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214741    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214744    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214746    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214749    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214751    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214754    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214756    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214759    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214762    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214764    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214767    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214770    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 10:53:26.216177 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214773    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214775    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214778    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214782    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214784    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214786    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214789    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214792    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214795    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214798    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214800    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214803    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214806    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214809    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214825    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214828    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214831    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.214834    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216444    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 10:53:26.216648 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216454    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216459    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216462    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216465    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216467    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216470    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216473    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216475    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216479    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216481    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216484    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216487    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216489    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216492    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216495    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216497    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216500    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216502    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216505    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 10:53:26.217157 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216508    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216511    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216513    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216516    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216519    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216521    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216524    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216527    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216530    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216533    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216535    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216538    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216541    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216543    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216547    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216550    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216554    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216557    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216559    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 10:53:26.217670 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216563    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216566    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216569    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216573    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216575    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216578    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216580    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216583    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216586    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216588    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216591    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216593    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216596    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216599    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216602    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216607    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216611    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216614    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216617    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216620    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 10:53:26.218146 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216623    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216626    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216629    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216631    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216634    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216637    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216639    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216642    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216644    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216647    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216650    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216652    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216655    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216658    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216660    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216664    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216667    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216669    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216672    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216674    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 10:53:26.218641 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216677    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216680    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216682    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216684    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216687    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216690    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216693    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216695    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216698    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216700    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216704    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216706    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216709    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216712    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216714    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216717    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216719    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216722    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216725    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.216728    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 10:53:26.219154 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217120    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217126    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217128    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217131    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217134    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217137    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217140    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217142    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217146    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217148    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217151    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217154    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217157    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217160    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217162    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217165    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217168    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217171    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217173    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217176    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 10:53:26.219645 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217178    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217181    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217186    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217189    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217191    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217194    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217197    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217200    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217202    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217205    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217207    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217210    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217213    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217216    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217219    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217222    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217224    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217227    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217230    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217232    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 10:53:26.220150 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217235    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217238    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217240    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217243    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217246    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217248    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217251    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217254    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217257    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217259    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217262    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217266    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217269    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217272    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217277    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217281    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217285    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217288    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217291    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 10:53:26.220615 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217294    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217296    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217299    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217302    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217304    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217307    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217310    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217312    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217315    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217317    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217320    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217323    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217327    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217331    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217334    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217337    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217340    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217343    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217346    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 10:53:26.221088 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217349    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217352    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217355    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217359    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217362    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217365    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217368    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217370    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217374    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217377    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217379    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217382    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217384    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217387    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217389    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217392    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217395    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217397    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217400    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217403    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 10:53:26.221538 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.217405    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217494    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217543    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217552    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217556    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217561    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217564    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217569    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217573    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217577    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217580    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217584    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217587    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217590    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217593    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217598    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217602    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217604    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217608    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217611    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217615    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217618    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217621    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217625    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 10:53:26.222025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217628    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217631    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217635    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217638    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217641    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217644    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217647    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217650    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217653    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217657    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217660    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217663    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217666    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217669    2562 flags.go:64] FLAG: --enable-server="true"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217672    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217677    2562 flags.go:64] FLAG: --event-burst="100"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217680    2562 flags.go:64] FLAG: --event-qps="50"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217683    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217687    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217690    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217694    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217697    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217700    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217704    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217707    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 10:53:26.222598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217710    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217713    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217716    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217719    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217722    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217725    2562 flags.go:64] FLAG: --feature-gates=""
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217732    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217735    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217738    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217742    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217745    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217748    2562 flags.go:64] FLAG: --help="false"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217751    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217754    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217757    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217760    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217763    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217767    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217770    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217772    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217775    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217778    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217781    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217784    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 10:53:26.223262 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217787    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217790    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217793    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217796    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217799    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217802    2562 flags.go:64] FLAG: --lock-file=""
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217806    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217809    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217822    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217828    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217831    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217835    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217838    2562 flags.go:64] FLAG: --logging-format="text"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217842    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217845    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217848    2562 flags.go:64] FLAG: --manifest-url=""
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217852    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217856    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217859    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217863    2562 flags.go:64] FLAG: --max-pods="110"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217866    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217870    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217872    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217876    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217879    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 10:53:26.223875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217882    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217885    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217893    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217896    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217899    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217902    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217905    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217911    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217914    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217917    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217927    2562 flags.go:64] FLAG: --port="10250"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217931    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217934    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0eef1a18a1abc52a0"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217938    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217941    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217945    2562 flags.go:64] FLAG: --register-node="true"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217948    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217956    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217959    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217962    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217964    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217968    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217971    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217974    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217977    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 10:53:26.224477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217980    2562 flags.go:64] FLAG: --runonce="false"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217983    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217986    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217989    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217992    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217995    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.217998    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218001    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218004    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218007    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218010    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218012    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218015    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218018    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218021    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218024    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218029    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218032    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218035    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218039    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218043    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218046    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218049    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218052    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218055    2562 flags.go:64] FLAG: --v="2"
Sep 17 10:53:26.225145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218059    2562 flags.go:64] FLAG: --version="false"
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218063    2562 flags.go:64] FLAG: --vmodule=""
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218066    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.218069    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218188    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218192    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218194    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218197    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218200    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218202    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218206    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218210    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218213    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218216    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218220    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218223    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218227    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218230    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218234    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 10:53:26.226041 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218237    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218240    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218243    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218246    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218249    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218252    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218256    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218260    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218264    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218267    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218269    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218272    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218274    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218277    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218280    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218283    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218285    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218288    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218290    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218293    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 10:53:26.226865 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218296    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218299    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218302    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218304    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218307    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218310    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218312    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218315    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218317    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218320    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218323    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218325    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218328    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218331    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218333    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218336    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218338    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218341    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218346    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218348    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 10:53:26.227783 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218353    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218355    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218358    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218361    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218363    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218366    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218375    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218378    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218380    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218383    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218386    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218389    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218392    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218395    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218397    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218400    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218403    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218405    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218408    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 10:53:26.228742 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218411    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218413    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218416    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218419    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218429    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218432    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218434    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218437    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218440    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218443    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218445    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218449    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218452    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218456    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218459    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218462    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218464    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218467    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218470    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218473    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 10:53:26.229596 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218475    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 10:53:26.230267 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218478    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 10:53:26.230267 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218481    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 10:53:26.230267 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218483    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 10:53:26.230267 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.218486    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 10:53:26.230267 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.219196    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 10:53:26.230267 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.228301    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 10:53:26.230267 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.228323    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 10:53:26.230267 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.228350    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 10:53:26.230267 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.228359    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 10:53:26.230267 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228451    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 10:53:26.230267 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228460    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 10:53:26.230267 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228468    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 10:53:26.230267 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228475    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 10:53:26.230267 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228480    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228485    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228489    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228494    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228498    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228502    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228509    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228514    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228520    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228524    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228528    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228533    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228537    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228541    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228545    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228550    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228554    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228558    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228562    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228567    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 10:53:26.230661 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228571    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228575    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228580    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228584    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228588    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228601    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228605    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228609    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228614    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228618    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228622    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228627    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228631    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228635    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228639    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228643    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228648    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228652    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228656    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 10:53:26.231490 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228660    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228664    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228668    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228673    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228677    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228681    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228685    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228689    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228693    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228697    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228701    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228705    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228710    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228714    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228718    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228722    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228726    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228730    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228735    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 10:53:26.232285 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228753    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228757    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228761    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228765    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228770    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228774    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228778    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228782    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228786    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228790    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228794    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228799    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228803    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228807    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228830    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228837    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228842    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228847    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228851    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 10:53:26.232788 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228855    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 10:53:26.233503 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228859    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 10:53:26.233503 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228864    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 10:53:26.233503 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228868    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 10:53:26.233503 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228872    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 10:53:26.233503 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228876    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 10:53:26.233503 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228880    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 10:53:26.233503 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228884    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 10:53:26.233503 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228888    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 10:53:26.233503 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228893    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 10:53:26.233503 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228898    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 10:53:26.233503 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228902    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 10:53:26.233503 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228907    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 10:53:26.233503 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228911    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 10:53:26.233503 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228926    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 10:53:26.233503 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228931    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 10:53:26.233503 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228935    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 10:53:26.233503 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.228939    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 10:53:26.233945 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.228947    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 10:53:26.233945 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229177    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 10:53:26.233945 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229187    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 10:53:26.233945 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229192    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 10:53:26.233945 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229196    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 10:53:26.233945 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229200    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 10:53:26.233945 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229204    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 10:53:26.233945 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229209    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 10:53:26.233945 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229212    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 10:53:26.233945 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229217    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 10:53:26.233945 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229221    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 10:53:26.233945 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229225    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 10:53:26.233945 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229229    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 10:53:26.233945 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229234    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 10:53:26.233945 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229238    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229242    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229246    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229250    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229254    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229266    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229271    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229275    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229280    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229284    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229289    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229293    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229297    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229302    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229306    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229318    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229322    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229327    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229332    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229336    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 10:53:26.234377 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229341    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229345    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229349    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229353    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229357    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229364    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229370    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229374    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229378    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229383    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229387    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229392    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229396    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229400    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229404    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229408    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229412    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229417    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229422    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229427    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 10:53:26.234895 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229431    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229435    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229440    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229444    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229448    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229453    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229457    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229461    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229473    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229478    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229483    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229487    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229491    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229496    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229499    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229504    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229508    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229512    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229516    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 10:53:26.235442 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229520    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229524    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229529    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229533    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229537    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229541    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229545    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229549    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229553    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229558    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229562    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229566    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229571    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229575    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229578    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229583    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229588    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229592    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229596    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229600    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 10:53:26.235917 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229604    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 10:53:26.236421 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229607    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 10:53:26.236421 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229621    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 10:53:26.236421 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229627    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 10:53:26.236421 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229632    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 10:53:26.236421 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229637    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 10:53:26.236421 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:26.229641    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 10:53:26.236421 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.229649    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 10:53:26.236421 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.229860    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 10:53:26.236421 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.233080    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 10:53:26.236421 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.234267    2562 server.go:1019] "Starting client certificate rotation"
Sep 17 10:53:26.236421 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.234399    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 10:53:26.236421 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.234456    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 10:53:26.237345 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.237328    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 10:53:26.245714 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.245686    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 10:53:26.261810 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.261784    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 17 10:53:26.264670 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.264651    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 10:53:26.266017 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.266003    2562 log.go:25] "Validated CRI v1 image API"
Sep 17 10:53:26.266941 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.266923    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 10:53:26.269584 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.269559    2562 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 90a6031b-9bab-4103-847c-0497b252d1c5:/dev/nvme0n1p4 f9f9d2e7-8805-44cc-aa69-78ca76f689d1:/dev/nvme0n1p3]
Sep 17 10:53:26.269641 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.269583    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 10:53:26.275163 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.275049    2562 manager.go:216] Machine: {Timestamp:2026-09-17 10:53:26.273653654 +0000 UTC m=+0.371699211 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099401 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f65e5168cfd39874e68d7e3e40938 SystemUUID:ec2f65e5-168c-fd39-874e-68d7e3e40938 BootID:c9d1391a-9b2e-470e-bcdc-e920f1c8be00 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e0:2b:28:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e0:2b:28:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:b1:d6:c2:d8:a6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 10:53:26.275163 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.275158    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 10:53:26.275275 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.275242    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 10:53:26.278701 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.278677    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 10:53:26.278858 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.278704    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-204.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 10:53:26.279595 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.279585    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 10:53:26.279625 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.279598    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 10:53:26.279625 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.279606    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 10:53:26.280463 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.280453    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 10:53:26.281730 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.281720    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 10:53:26.286761 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.286749    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 10:53:26.286881 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.286872    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 10:53:26.286914 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.286903    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 10:53:26.286914 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.286913    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 10:53:26.286977 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.286925    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 10:53:26.286977 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.286935    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 17 10:53:26.286977 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.286945    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 10:53:26.287320 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.287298    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 10:53:26.287412 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.287336    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 10:53:26.288449 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.288423    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jclrq"
Sep 17 10:53:26.289760 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.289741    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 17 10:53:26.292714 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.292696    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 10:53:26.292806 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.292735    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 10:53:26.295640 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.295625    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 10:53:26.295727 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.295647    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 10:53:26.295727 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.295656    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 10:53:26.295727 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.295664    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 10:53:26.295727 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.295686    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 10:53:26.295727 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.295695    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 10:53:26.295727 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.295715    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 10:53:26.295727 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.295723    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 10:53:26.295969 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.295733    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 10:53:26.295969 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.295742    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 10:53:26.295969 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.295759    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 10:53:26.295969 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.295768    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 10:53:26.296900 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.296882    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 10:53:26.296900 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.296900    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 10:53:26.297073 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.297056    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jclrq"
Sep 17 10:53:26.298225 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:26.298041    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-204.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 10:53:26.298225 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:26.298041    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 10:53:26.298319 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.298245    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-204.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 10:53:26.300789 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.300775    2562 server.go:1268] "Started kubelet"
Sep 17 10:53:26.300899 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.300871    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 10:53:26.305036 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.305014    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 10:53:26.305481 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.300884    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 10:53:26.305638 ip-10-0-139-204 systemd[1]: Started Kubernetes Kubelet.
Sep 17 10:53:26.307065 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.307047    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 10:53:26.309452 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.309435    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 10:53:26.310193 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:26.310173    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 10:53:26.312429 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:26.311343    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-204.ec2.internal.18d616495e880703  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-204.ec2.internal,UID:ip-10-0-139-204.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-204.ec2.internal,},FirstTimestamp:2026-09-17 10:53:26.300747523 +0000 UTC m=+0.398793080,LastTimestamp:2026-09-17 10:53:26.300747523 +0000 UTC m=+0.398793080,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-204.ec2.internal,}"
Sep 17 10:53:26.315551 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.315531    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 10:53:26.316273 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.316259    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 10:53:26.316372 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:26.316352    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found"
Sep 17 10:53:26.317009 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.316992    2562 factory.go:55] Registering systemd factory
Sep 17 10:53:26.317009 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.317009    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 17 10:53:26.317144 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.317027    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 10:53:26.317144 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.317033    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 10:53:26.317144 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.317048    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 10:53:26.317144 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.317121    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 10:53:26.317144 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.317130    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 10:53:26.317348 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.317229    2562 factory.go:153] Registering CRI-O factory
Sep 17 10:53:26.317348 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.317239    2562 factory.go:223] Registration of the crio container factory successfully
Sep 17 10:53:26.317348 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.317285    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 10:53:26.317348 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.317298    2562 factory.go:103] Registering Raw factory
Sep 17 10:53:26.317348 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.317308    2562 manager.go:1196] Started watching for new ooms in manager
Sep 17 10:53:26.317790 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.317776    2562 manager.go:349] Starting recovery of all containers
Sep 17 10:53:26.329141 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.329124    2562 manager.go:354] Recovery completed
Sep 17 10:53:26.329621 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.329605    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 10:53:26.333604 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:26.333588    2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-204.ec2.internal\" not found" node="ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.334332 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.334321    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 10:53:26.336268 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.336254    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 10:53:26.336341 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.336283    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 10:53:26.336341 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.336296    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientPID"
Sep 17 10:53:26.336840 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.336809    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 10:53:26.336903 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.336843    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 10:53:26.336903 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.336865    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 10:53:26.336903 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.336834    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 10:53:26.337979 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.337966    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 10:53:26.338057 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.337994    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 10:53:26.338057 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.338005    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientPID"
Sep 17 10:53:26.339342 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.339330    2562 policy_none.go:49] "None policy: Start"
Sep 17 10:53:26.339402 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.339347    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 10:53:26.339402 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.339357    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 10:53:26.341440 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.341430    2562 policy_none.go:47] "Start"
Sep 17 10:53:26.378424 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.378405    2562 manager.go:337] "Starting Device Plugin manager"
Sep 17 10:53:26.395486 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:26.378452    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 10:53:26.395486 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.378465    2562 server.go:87] "Starting device plugin registration server"
Sep 17 10:53:26.395486 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.378857    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 10:53:26.395486 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.378869    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 10:53:26.395486 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.379008    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 10:53:26.395486 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.379090    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 10:53:26.395486 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.379098    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 10:53:26.395486 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:26.379685    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 10:53:26.395486 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:26.379739    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-204.ec2.internal\" not found"
Sep 17 10:53:26.447679 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.447643    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 10:53:26.448829 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.448797    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 10:53:26.448942 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.448870    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 10:53:26.448942 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.448902    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 10:53:26.449040 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:26.449010    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 10:53:26.451547 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.451528    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 10:53:26.479804 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.479706    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 10:53:26.480670 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.480655    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 10:53:26.480765 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.480688    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 10:53:26.480765 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.480703    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientPID"
Sep 17 10:53:26.480765 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.480736    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.488943 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.488923    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.488995 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:26.488947    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-204.ec2.internal\": node \"ip-10-0-139-204.ec2.internal\" not found"
Sep 17 10:53:26.512101 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:26.512070    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found"
Sep 17 10:53:26.549345 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.549312    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal"]
Sep 17 10:53:26.549478 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.549406    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 10:53:26.550925 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.550907    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 10:53:26.551025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.550941    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 10:53:26.551025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.550956    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientPID"
Sep 17 10:53:26.553294 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.553279    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 10:53:26.553416 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.553393    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.553468 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.553432    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 10:53:26.554069 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.554053    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 10:53:26.554151 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.554080    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 10:53:26.554151 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.554054    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 10:53:26.554151 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.554090    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientPID"
Sep 17 10:53:26.554151 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.554109    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 10:53:26.554151 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.554125    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientPID"
Sep 17 10:53:26.556316 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.556302    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.556372 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.556335    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 10:53:26.556996 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.556977    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 10:53:26.557092 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.557013    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 10:53:26.557092 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.557025    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientPID"
Sep 17 10:53:26.587397 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:26.587373    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-204.ec2.internal\" not found" node="ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.591739 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:26.591723    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-204.ec2.internal\" not found" node="ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.612386 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:26.612358    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found"
Sep 17 10:53:26.618941 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.618919    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d219867a7ad5b4e9ab4856e1757c643d-config\") pod \"kube-apiserver-proxy-ip-10-0-139-204.ec2.internal\" (UID: \"d219867a7ad5b4e9ab4856e1757c643d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.618941 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.618943    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/240331bd5dd4e3d928b56e96a0631dd5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" (UID: \"240331bd5dd4e3d928b56e96a0631dd5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.619073 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.618962    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/240331bd5dd4e3d928b56e96a0631dd5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" (UID: \"240331bd5dd4e3d928b56e96a0631dd5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.712961 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:26.712931    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found"
Sep 17 10:53:26.719271 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.719249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/240331bd5dd4e3d928b56e96a0631dd5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" (UID: \"240331bd5dd4e3d928b56e96a0631dd5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.719386 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.719279    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/240331bd5dd4e3d928b56e96a0631dd5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" (UID: \"240331bd5dd4e3d928b56e96a0631dd5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.719386 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.719303    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d219867a7ad5b4e9ab4856e1757c643d-config\") pod \"kube-apiserver-proxy-ip-10-0-139-204.ec2.internal\" (UID: \"d219867a7ad5b4e9ab4856e1757c643d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.719386 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.719359    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/240331bd5dd4e3d928b56e96a0631dd5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" (UID: \"240331bd5dd4e3d928b56e96a0631dd5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.719482 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.719379    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/240331bd5dd4e3d928b56e96a0631dd5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" (UID: \"240331bd5dd4e3d928b56e96a0631dd5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.719482 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.719419    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d219867a7ad5b4e9ab4856e1757c643d-config\") pod \"kube-apiserver-proxy-ip-10-0-139-204.ec2.internal\" (UID: \"d219867a7ad5b4e9ab4856e1757c643d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.813781 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:26.813691    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found"
Sep 17 10:53:26.890231 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.890200    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.893883 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:26.893853    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal"
Sep 17 10:53:26.914774 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:26.914745    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found"
Sep 17 10:53:27.015357 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:27.015318    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found"
Sep 17 10:53:27.115839 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:27.115783    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found"
Sep 17 10:53:27.223233 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:27.223188    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found"
Sep 17 10:53:27.234453 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.234417    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 10:53:27.234595 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.234574    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 10:53:27.234643 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.234613    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 10:53:27.264196 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.264162    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 10:53:27.287966 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.287943    2562 apiserver.go:52] "Watching apiserver"
Sep 17 10:53:27.299449 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.299423    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 10:53:27.301139 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.301118    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-flklx","openshift-dns/node-resolver-q5vb8","openshift-multus/multus-additional-cni-plugins-thr8q","openshift-multus/network-metrics-daemon-k6wfk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq","openshift-image-registry/node-ca-hlqjq","openshift-multus/multus-cbwwt","openshift-network-diagnostics/network-check-target-tqgcf","openshift-network-operator/iptables-alerter-jqhht","openshift-ovn-kubernetes/ovnkube-node-lzn6w","kube-system/global-pull-secret-syncer-csvxz","kube-system/konnectivity-agent-8gz59"]
Sep 17 10:53:27.303880 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.303862    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.306040 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.306010    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 10:48:26 +0000 UTC" deadline="2028-02-13 05:34:44.545219892 +0000 UTC"
Sep 17 10:53:27.306116 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.306039    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12330h41m17.239184622s"
Sep 17 10:53:27.306194 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.306179    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q5vb8"
Sep 17 10:53:27.306728 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.306711    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gxh4j\""
Sep 17 10:53:27.306813 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.306763    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 10:53:27.306885 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.306870    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 10:53:27.308606 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.308590    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.309000 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.308986    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 10:53:27.309076 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.309059    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 10:53:27.309294 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.309280    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7g47n\""
Sep 17 10:53:27.310955 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.310938    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:27.311034 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:27.311013    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6wfk" podUID="96dd2d24-1a37-4504-b738-19be03267b6a"
Sep 17 10:53:27.311402 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.311388    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 10:53:27.312446 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.312421    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 10:53:27.312560 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.312544    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 10:53:27.313248 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.313051    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 10:53:27.313248 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.313135    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 10:53:27.313248 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.313199    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fmzpj\""
Sep 17 10:53:27.314579 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.313654    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.316063 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.316045    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 10:53:27.316187 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.316171    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hlqjq"
Sep 17 10:53:27.316452 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.316434    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal"
Sep 17 10:53:27.316452 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.316444    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 10:53:27.317006 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.316992    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 10:53:27.317284 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.317270    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 10:53:27.317396 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.317380    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-96p5l\""
Sep 17 10:53:27.318535 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.318518    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.318978 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.318962    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 10:53:27.319840 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.319819    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rb9z7\""
Sep 17 10:53:27.319953 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.319886    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 10:53:27.319953 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.319913    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 10:53:27.320856 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.320839    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:27.320934 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:27.320908    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgcf" podUID="9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5"
Sep 17 10:53:27.321635 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.321617    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 10:53:27.321785 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.321770    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-98pzb\""
Sep 17 10:53:27.323333 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323314    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jqhht"
Sep 17 10:53:27.323401 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323361    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgd6h\" (UniqueName: \"kubernetes.io/projected/7e7c0e7e-ab26-4238-b048-869055b359b2-kube-api-access-lgd6h\") pod \"node-ca-hlqjq\" (UID: \"7e7c0e7e-ab26-4238-b048-869055b359b2\") " pod="openshift-image-registry/node-ca-hlqjq"
Sep 17 10:53:27.323401 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323393    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-system-cni-dir\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.323535 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323418    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-hostroot\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.323535 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323441    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-multus-conf-dir\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.323535 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323493    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16d6daf4-f47b-4282-8b60-e752eb65e772-hosts-file\") pod \"node-resolver-q5vb8\" (UID: \"16d6daf4-f47b-4282-8b60-e752eb65e772\") " pod="openshift-dns/node-resolver-q5vb8"
Sep 17 10:53:27.323696 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323569    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-etc-sysctl-d\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.323696 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323603    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-sys-fs\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.323696 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323626    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e7c0e7e-ab26-4238-b048-869055b359b2-host\") pod \"node-ca-hlqjq\" (UID: \"7e7c0e7e-ab26-4238-b048-869055b359b2\") " pod="openshift-image-registry/node-ca-hlqjq"
Sep 17 10:53:27.323696 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323659    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-host-var-lib-cni-multus\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.323696 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323684    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-host-var-lib-kubelet\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.323977 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323722    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-etc-kubernetes\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.323977 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323744    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdkxh\" (UniqueName: \"kubernetes.io/projected/8778bc91-143b-4854-8393-9e502febb3f5-kube-api-access-hdkxh\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.323977 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323760    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d6daf4-f47b-4282-8b60-e752eb65e772-tmp-dir\") pod \"node-resolver-q5vb8\" (UID: \"16d6daf4-f47b-4282-8b60-e752eb65e772\") " pod="openshift-dns/node-resolver-q5vb8"
Sep 17 10:53:27.323977 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323794    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs\") pod \"network-metrics-daemon-k6wfk\" (UID: \"96dd2d24-1a37-4504-b738-19be03267b6a\") " pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:27.323977 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323836    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-socket-dir\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.323977 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323864    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-etc-kubernetes\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.323977 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323887    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-run\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.323977 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323910    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-var-lib-kubelet\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.323977 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323952    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-device-dir\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.324445 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.323995    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjjdr\" (UniqueName: \"kubernetes.io/projected/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-kube-api-access-zjjdr\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.324445 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324022    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e7c0e7e-ab26-4238-b048-869055b359b2-serviceca\") pod \"node-ca-hlqjq\" (UID: \"7e7c0e7e-ab26-4238-b048-869055b359b2\") " pod="openshift-image-registry/node-ca-hlqjq"
Sep 17 10:53:27.324445 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324045    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03e1a707-7a61-4d14-bf87-73bb46d021cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.324445 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324089    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-etc-sysctl-conf\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.324445 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324115    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03e1a707-7a61-4d14-bf87-73bb46d021cd-system-cni-dir\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.324445 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324155    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-multus-cni-dir\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.324445 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-cnibin\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.324445 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324218    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-multus-socket-dir-parent\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.324445 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324241    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-host-var-lib-cni-bin\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.324445 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324265    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-etc-sysconfig\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.324445 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324305    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03e1a707-7a61-4d14-bf87-73bb46d021cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.324445 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324366    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03e1a707-7a61-4d14-bf87-73bb46d021cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.324445 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324398    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8778bc91-143b-4854-8393-9e502febb3f5-multus-daemon-config\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.324445 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324429    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e926f447-9657-48f4-a086-965984dabceb-tmp\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.325020 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324454    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-host-run-multus-certs\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.325020 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324480    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5rjd\" (UniqueName: \"kubernetes.io/projected/16d6daf4-f47b-4282-8b60-e752eb65e772-kube-api-access-h5rjd\") pod \"node-resolver-q5vb8\" (UID: \"16d6daf4-f47b-4282-8b60-e752eb65e772\") " pod="openshift-dns/node-resolver-q5vb8"
Sep 17 10:53:27.325020 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324507    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zss8z\" (UniqueName: \"kubernetes.io/projected/96dd2d24-1a37-4504-b738-19be03267b6a-kube-api-access-zss8z\") pod \"network-metrics-daemon-k6wfk\" (UID: \"96dd2d24-1a37-4504-b738-19be03267b6a\") " pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:27.325020 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324533    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-etc-modprobe-d\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.325020 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324556    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-sys\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.325020 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324579    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-lib-modules\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.325020 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324602    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-host\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.325020 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324625    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqt8f\" (UniqueName: \"kubernetes.io/projected/e926f447-9657-48f4-a086-965984dabceb-kube-api-access-lqt8f\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.325020 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324651    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03e1a707-7a61-4d14-bf87-73bb46d021cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.325020 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324676    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.325020 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324708    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-etc-systemd\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.325020 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324729    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03e1a707-7a61-4d14-bf87-73bb46d021cd-cnibin\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.325020 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324761    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-registration-dir\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.325020 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324777    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8778bc91-143b-4854-8393-9e502febb3f5-cni-binary-copy\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.325020 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324793    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-host-run-k8s-cni-cncf-io\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.325020 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324808    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-host-run-netns\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.325545 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324830    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e926f447-9657-48f4-a086-965984dabceb-etc-tuned\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.325545 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324862    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4l8g\" (UniqueName: \"kubernetes.io/projected/03e1a707-7a61-4d14-bf87-73bb46d021cd-kube-api-access-s4l8g\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.325545 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.324884    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.325799 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.325785    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.326735 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.326717    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jbb5p\""
Sep 17 10:53:27.326841 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.326826    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 10:53:27.327244 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.327230    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 10:53:27.327534 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.327520    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 10:53:27.328436 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.328406    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:27.328523 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:27.328485    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csvxz" podUID="df19ed65-1e04-4637-b6d6-8879cd6b750c"
Sep 17 10:53:27.330072 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.330056    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 10:53:27.330388 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.330361    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 10:53:27.330388 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.330361    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kjhsk\""
Sep 17 10:53:27.330630 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.330616    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 10:53:27.330679 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.330617    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 10:53:27.330679 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.330656    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 10:53:27.330881 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.330866    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8gz59"
Sep 17 10:53:27.330969 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.330946    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 10:53:27.334190 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.334170    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 10:53:27.334288 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.334272    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal"
Sep 17 10:53:27.334422 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.334408    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 10:53:27.334684 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.334665    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal"]
Sep 17 10:53:27.334751 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.334692    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 10:53:27.335108 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.335093    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5brkj\""
Sep 17 10:53:27.342684 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.342666    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 10:53:27.345378 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.345363    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal"]
Sep 17 10:53:27.345806 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.345785    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 10:53:27.372665 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.372643    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7jvvw"
Sep 17 10:53:27.387997 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.387978    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7jvvw"
Sep 17 10:53:27.418336 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.418315    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 10:53:27.425903 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.425883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-etc-sysctl-conf\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.425978 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.425915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-multus-cni-dir\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.425978 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.425948    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x898x\" (UniqueName: \"kubernetes.io/projected/54753226-3238-4827-a57c-249e9bb251f8-kube-api-access-x898x\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.426067 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.425975    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03e1a707-7a61-4d14-bf87-73bb46d021cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.426067 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426005    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03e1a707-7a61-4d14-bf87-73bb46d021cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.426067 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426010    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-multus-cni-dir\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.426067 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426013    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-etc-sysctl-conf\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.426067 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426030    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-run-ovn\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.426067 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426057    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zss8z\" (UniqueName: \"kubernetes.io/projected/96dd2d24-1a37-4504-b738-19be03267b6a-kube-api-access-zss8z\") pod \"network-metrics-daemon-k6wfk\" (UID: \"96dd2d24-1a37-4504-b738-19be03267b6a\") " pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:27.426317 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-etc-modprobe-d\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.426317 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426121    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-sys\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.426317 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-lib-modules\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.426317 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426169    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqt8f\" (UniqueName: \"kubernetes.io/projected/e926f447-9657-48f4-a086-965984dabceb-kube-api-access-lqt8f\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.426317 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426212    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03e1a707-7a61-4d14-bf87-73bb46d021cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.426317 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426236    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-etc-modprobe-d\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.426317 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426288    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-sys\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.426317 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426295    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.426317 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426240    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.426842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426338    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8778bc91-143b-4854-8393-9e502febb3f5-cni-binary-copy\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.426842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03e1a707-7a61-4d14-bf87-73bb46d021cd-cnibin\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.426842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426420    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-lib-modules\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.426842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426469    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-registration-dir\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.426842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426502    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-host-run-netns\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.426842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426524    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16d6daf4-f47b-4282-8b60-e752eb65e772-hosts-file\") pod \"node-resolver-q5vb8\" (UID: \"16d6daf4-f47b-4282-8b60-e752eb65e772\") " pod="openshift-dns/node-resolver-q5vb8"
Sep 17 10:53:27.426842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426565    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d6daf4-f47b-4282-8b60-e752eb65e772-tmp-dir\") pod \"node-resolver-q5vb8\" (UID: \"16d6daf4-f47b-4282-8b60-e752eb65e772\") " pod="openshift-dns/node-resolver-q5vb8"
Sep 17 10:53:27.426842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426592    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-log-socket\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.426842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426616    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-cni-bin\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.426842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426624    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03e1a707-7a61-4d14-bf87-73bb46d021cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.426842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426645    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4l8g\" (UniqueName: \"kubernetes.io/projected/03e1a707-7a61-4d14-bf87-73bb46d021cd-kube-api-access-s4l8g\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.426842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426672    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgd6h\" (UniqueName: \"kubernetes.io/projected/7e7c0e7e-ab26-4238-b048-869055b359b2-kube-api-access-lgd6h\") pod \"node-ca-hlqjq\" (UID: \"7e7c0e7e-ab26-4238-b048-869055b359b2\") " pod="openshift-image-registry/node-ca-hlqjq"
Sep 17 10:53:27.426842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426696    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4efd824-b4e0-4343-9085-553f8e965648-agent-certs\") pod \"konnectivity-agent-8gz59\" (UID: \"c4efd824-b4e0-4343-9085-553f8e965648\") " pod="kube-system/konnectivity-agent-8gz59"
Sep 17 10:53:27.426842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426703    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03e1a707-7a61-4d14-bf87-73bb46d021cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.426842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426722    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/df19ed65-1e04-4637-b6d6-8879cd6b750c-dbus\") pod \"global-pull-secret-syncer-csvxz\" (UID: \"df19ed65-1e04-4637-b6d6-8879cd6b750c\") " pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:27.426842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426769    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9jd8\" (UniqueName: \"kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8\") pod \"network-check-target-tqgcf\" (UID: \"9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5\") " pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:27.426842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426801    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-systemd-units\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.427658 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426829    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/54753226-3238-4827-a57c-249e9bb251f8-env-overrides\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.427658 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426845    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03e1a707-7a61-4d14-bf87-73bb46d021cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.427658 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426855    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret\") pod \"global-pull-secret-syncer-csvxz\" (UID: \"df19ed65-1e04-4637-b6d6-8879cd6b750c\") " pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:27.427658 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426912    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs\") pod \"network-metrics-daemon-k6wfk\" (UID: \"96dd2d24-1a37-4504-b738-19be03267b6a\") " pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:27.427658 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426925    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-registration-dir\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.427658 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426944    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e7c0e7e-ab26-4238-b048-869055b359b2-serviceca\") pod \"node-ca-hlqjq\" (UID: \"7e7c0e7e-ab26-4238-b048-869055b359b2\") " pod="openshift-image-registry/node-ca-hlqjq"
Sep 17 10:53:27.427658 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426947    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8778bc91-143b-4854-8393-9e502febb3f5-cni-binary-copy\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.427658 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426967    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03e1a707-7a61-4d14-bf87-73bb46d021cd-cnibin\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.427658 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426973    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03e1a707-7a61-4d14-bf87-73bb46d021cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.427658 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.426984    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d6daf4-f47b-4282-8b60-e752eb65e772-tmp-dir\") pod \"node-resolver-q5vb8\" (UID: \"16d6daf4-f47b-4282-8b60-e752eb65e772\") " pod="openshift-dns/node-resolver-q5vb8"
Sep 17 10:53:27.427658 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427007    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03e1a707-7a61-4d14-bf87-73bb46d021cd-system-cni-dir\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.427658 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427056    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16d6daf4-f47b-4282-8b60-e752eb65e772-hosts-file\") pod \"node-resolver-q5vb8\" (UID: \"16d6daf4-f47b-4282-8b60-e752eb65e772\") " pod="openshift-dns/node-resolver-q5vb8"
Sep 17 10:53:27.427658 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427081    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-cnibin\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.427658 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427084    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-host-run-netns\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.427658 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427109    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-host-var-lib-cni-bin\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.427658 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427123    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03e1a707-7a61-4d14-bf87-73bb46d021cd-system-cni-dir\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.427658 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427183    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-slash\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.427658 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:27.427205    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 10:53:27.428596 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427196    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-cnibin\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.428596 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427236    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-var-lib-openvswitch\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.428596 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427270    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-etc-sysconfig\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.428596 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:27.427295    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs podName:96dd2d24-1a37-4504-b738-19be03267b6a nodeName:}" failed. No retries permitted until 2026-09-17 10:53:27.927271222 +0000 UTC m=+2.025316790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs") pod "network-metrics-daemon-k6wfk" (UID: "96dd2d24-1a37-4504-b738-19be03267b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 10:53:27.428596 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427303    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-host-var-lib-cni-bin\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.428596 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427353    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8778bc91-143b-4854-8393-9e502febb3f5-multus-daemon-config\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.428596 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427403    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-etc-sysconfig\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.428596 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427453    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e7c0e7e-ab26-4238-b048-869055b359b2-serviceca\") pod \"node-ca-hlqjq\" (UID: \"7e7c0e7e-ab26-4238-b048-869055b359b2\") " pod="openshift-image-registry/node-ca-hlqjq"
Sep 17 10:53:27.428596 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427508    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4efd824-b4e0-4343-9085-553f8e965648-konnectivity-ca\") pod \"konnectivity-agent-8gz59\" (UID: \"c4efd824-b4e0-4343-9085-553f8e965648\") " pod="kube-system/konnectivity-agent-8gz59"
Sep 17 10:53:27.428596 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427554    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dcb375fa-358a-4aef-9630-f6bf9c46120b-iptables-alerter-script\") pod \"iptables-alerter-jqhht\" (UID: \"dcb375fa-358a-4aef-9630-f6bf9c46120b\") " pod="openshift-network-operator/iptables-alerter-jqhht"
Sep 17 10:53:27.428596 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427602    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03e1a707-7a61-4d14-bf87-73bb46d021cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.428596 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427606    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-kubelet\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.428596 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427657    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-etc-openvswitch\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.428596 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427677    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/54753226-3238-4827-a57c-249e9bb251f8-ovnkube-config\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.428596 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427710    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e926f447-9657-48f4-a086-965984dabceb-tmp\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.428596 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427732    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-host-run-multus-certs\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.429283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427748    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcb375fa-358a-4aef-9630-f6bf9c46120b-host-slash\") pod \"iptables-alerter-jqhht\" (UID: \"dcb375fa-358a-4aef-9630-f6bf9c46120b\") " pod="openshift-network-operator/iptables-alerter-jqhht"
Sep 17 10:53:27.429283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427764    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/54753226-3238-4827-a57c-249e9bb251f8-ovn-node-metrics-cert\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.429283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427783    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8778bc91-143b-4854-8393-9e502febb3f5-multus-daemon-config\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.429283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427783    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-host\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.429283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427848    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-host-run-multus-certs\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.429283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-host-run-k8s-cni-cncf-io\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.429283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427916    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-host\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.429283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.427962    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-host-var-lib-kubelet\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.429283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428039    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-host-run-k8s-cni-cncf-io\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.429283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428071    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-etc-kubernetes\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.429283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428086    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 10:53:27.429283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdkxh\" (UniqueName: \"kubernetes.io/projected/8778bc91-143b-4854-8393-9e502febb3f5-kube-api-access-hdkxh\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.429283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428094    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-host-var-lib-kubelet\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.429283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428115    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4z4m\" (UniqueName: \"kubernetes.io/projected/dcb375fa-358a-4aef-9630-f6bf9c46120b-kube-api-access-s4z4m\") pod \"iptables-alerter-jqhht\" (UID: \"dcb375fa-358a-4aef-9630-f6bf9c46120b\") " pod="openshift-network-operator/iptables-alerter-jqhht"
Sep 17 10:53:27.429283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428152    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.429283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428154    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-etc-kubernetes\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.429283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.429283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428219    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-etc-systemd\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.429864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428245    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-hostroot\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.429864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428263    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-etc-systemd\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.429864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428264    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-multus-conf-dir\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.429864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428291    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-multus-conf-dir\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.429864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428299    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-hostroot\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.429864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428307    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-run-netns\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.429864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428333    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-run-systemd\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.429864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e926f447-9657-48f4-a086-965984dabceb-etc-tuned\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.429864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428396    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.429864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428443    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.429864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428472    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-system-cni-dir\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.429864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428496    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-run-openvswitch\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.429864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428525    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-cni-netd\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.429864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428542    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-etc-sysctl-d\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.429864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428558    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-sys-fs\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.429864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428569    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-system-cni-dir\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.429864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428578    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e7c0e7e-ab26-4238-b048-869055b359b2-host\") pod \"node-ca-hlqjq\" (UID: \"7e7c0e7e-ab26-4238-b048-869055b359b2\") " pod="openshift-image-registry/node-ca-hlqjq"
Sep 17 10:53:27.429864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428615    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e7c0e7e-ab26-4238-b048-869055b359b2-host\") pod \"node-ca-hlqjq\" (UID: \"7e7c0e7e-ab26-4238-b048-869055b359b2\") " pod="openshift-image-registry/node-ca-hlqjq"
Sep 17 10:53:27.430458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428625    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-host-var-lib-cni-multus\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.430458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428648    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-node-log\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.430458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428640    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-sys-fs\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.430458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428669    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/54753226-3238-4827-a57c-249e9bb251f8-ovnkube-script-lib\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.430458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428689    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-host-var-lib-cni-multus\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.430458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428697    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/df19ed65-1e04-4637-b6d6-8879cd6b750c-kubelet-config\") pod \"global-pull-secret-syncer-csvxz\" (UID: \"df19ed65-1e04-4637-b6d6-8879cd6b750c\") " pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:27.430458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428726    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rjd\" (UniqueName: \"kubernetes.io/projected/16d6daf4-f47b-4282-8b60-e752eb65e772-kube-api-access-h5rjd\") pod \"node-resolver-q5vb8\" (UID: \"16d6daf4-f47b-4282-8b60-e752eb65e772\") " pod="openshift-dns/node-resolver-q5vb8"
Sep 17 10:53:27.430458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428742    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-socket-dir\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.430458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428726    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-etc-sysctl-d\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.430458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428836    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-socket-dir\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.430458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428850    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-etc-kubernetes\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.430458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428891    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-run\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.430458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-var-lib-kubelet\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.430458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428892    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-etc-kubernetes\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.430458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428930    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-device-dir\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.430458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428953    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-run\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.430458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428968    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-device-dir\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.431080 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjjdr\" (UniqueName: \"kubernetes.io/projected/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-kube-api-access-zjjdr\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.431080 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.428991    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e926f447-9657-48f4-a086-965984dabceb-var-lib-kubelet\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.431080 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.429006    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-multus-socket-dir-parent\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.431080 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.429047    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8778bc91-143b-4854-8393-9e502febb3f5-multus-socket-dir-parent\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.431387 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.431371    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e926f447-9657-48f4-a086-965984dabceb-tmp\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.431493 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.431480    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e926f447-9657-48f4-a086-965984dabceb-etc-tuned\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.439187 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.439160    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgd6h\" (UniqueName: \"kubernetes.io/projected/7e7c0e7e-ab26-4238-b048-869055b359b2-kube-api-access-lgd6h\") pod \"node-ca-hlqjq\" (UID: \"7e7c0e7e-ab26-4238-b048-869055b359b2\") " pod="openshift-image-registry/node-ca-hlqjq"
Sep 17 10:53:27.439955 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.439912    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zss8z\" (UniqueName: \"kubernetes.io/projected/96dd2d24-1a37-4504-b738-19be03267b6a-kube-api-access-zss8z\") pod \"network-metrics-daemon-k6wfk\" (UID: \"96dd2d24-1a37-4504-b738-19be03267b6a\") " pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:27.440233 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.440210    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5rjd\" (UniqueName: \"kubernetes.io/projected/16d6daf4-f47b-4282-8b60-e752eb65e772-kube-api-access-h5rjd\") pod \"node-resolver-q5vb8\" (UID: \"16d6daf4-f47b-4282-8b60-e752eb65e772\") " pod="openshift-dns/node-resolver-q5vb8"
Sep 17 10:53:27.440430 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.440410    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4l8g\" (UniqueName: \"kubernetes.io/projected/03e1a707-7a61-4d14-bf87-73bb46d021cd-kube-api-access-s4l8g\") pod \"multus-additional-cni-plugins-thr8q\" (UID: \"03e1a707-7a61-4d14-bf87-73bb46d021cd\") " pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.440517 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.440462    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqt8f\" (UniqueName: \"kubernetes.io/projected/e926f447-9657-48f4-a086-965984dabceb-kube-api-access-lqt8f\") pod \"tuned-flklx\" (UID: \"e926f447-9657-48f4-a086-965984dabceb\") " pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.440760 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.440738    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjjdr\" (UniqueName: \"kubernetes.io/projected/57bf4eda-96bf-428d-8d4e-9ad181d5a8d1-kube-api-access-zjjdr\") pod \"aws-ebs-csi-driver-node-jz8tq\" (UID: \"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.441059 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.441040    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdkxh\" (UniqueName: \"kubernetes.io/projected/8778bc91-143b-4854-8393-9e502febb3f5-kube-api-access-hdkxh\") pod \"multus-cbwwt\" (UID: \"8778bc91-143b-4854-8393-9e502febb3f5\") " pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.529276 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529244    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/54753226-3238-4827-a57c-249e9bb251f8-ovnkube-config\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.529276 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529280    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcb375fa-358a-4aef-9630-f6bf9c46120b-host-slash\") pod \"iptables-alerter-jqhht\" (UID: \"dcb375fa-358a-4aef-9630-f6bf9c46120b\") " pod="openshift-network-operator/iptables-alerter-jqhht"
Sep 17 10:53:27.529584 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529299    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/54753226-3238-4827-a57c-249e9bb251f8-ovn-node-metrics-cert\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.529584 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4z4m\" (UniqueName: \"kubernetes.io/projected/dcb375fa-358a-4aef-9630-f6bf9c46120b-kube-api-access-s4z4m\") pod \"iptables-alerter-jqhht\" (UID: \"dcb375fa-358a-4aef-9630-f6bf9c46120b\") " pod="openshift-network-operator/iptables-alerter-jqhht"
Sep 17 10:53:27.529584 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529344    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.529584 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.529584 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529398    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcb375fa-358a-4aef-9630-f6bf9c46120b-host-slash\") pod \"iptables-alerter-jqhht\" (UID: \"dcb375fa-358a-4aef-9630-f6bf9c46120b\") " pod="openshift-network-operator/iptables-alerter-jqhht"
Sep 17 10:53:27.529584 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529409    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-run-netns\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.529584 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529461    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-run-systemd\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.529584 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529472    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.529584 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529490    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-run-openvswitch\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.529584 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529463    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-run-netns\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.529584 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529521    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.529584 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529553    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-run-systemd\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.529584 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529568    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-run-openvswitch\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.530157 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529596    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-cni-netd\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.530157 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529648    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-node-log\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.530157 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529676    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/54753226-3238-4827-a57c-249e9bb251f8-ovnkube-script-lib\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.530157 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529691    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-node-log\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.530157 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529687    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-cni-netd\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.530157 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529703    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/df19ed65-1e04-4637-b6d6-8879cd6b750c-kubelet-config\") pod \"global-pull-secret-syncer-csvxz\" (UID: \"df19ed65-1e04-4637-b6d6-8879cd6b750c\") " pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:27.530157 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529756    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/df19ed65-1e04-4637-b6d6-8879cd6b750c-kubelet-config\") pod \"global-pull-secret-syncer-csvxz\" (UID: \"df19ed65-1e04-4637-b6d6-8879cd6b750c\") " pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:27.530157 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529771    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x898x\" (UniqueName: \"kubernetes.io/projected/54753226-3238-4827-a57c-249e9bb251f8-kube-api-access-x898x\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.530157 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529802    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-run-ovn\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.530157 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529881    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-run-ovn\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.530157 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.529950    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/54753226-3238-4827-a57c-249e9bb251f8-ovnkube-config\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.530157 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-log-socket\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.530157 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530060    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-cni-bin\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.530157 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530094    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4efd824-b4e0-4343-9085-553f8e965648-agent-certs\") pod \"konnectivity-agent-8gz59\" (UID: \"c4efd824-b4e0-4343-9085-553f8e965648\") " pod="kube-system/konnectivity-agent-8gz59"
Sep 17 10:53:27.530157 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530102    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-log-socket\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.530157 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530120    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/df19ed65-1e04-4637-b6d6-8879cd6b750c-dbus\") pod \"global-pull-secret-syncer-csvxz\" (UID: \"df19ed65-1e04-4637-b6d6-8879cd6b750c\") " pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:27.530157 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530148    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jd8\" (UniqueName: \"kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8\") pod \"network-check-target-tqgcf\" (UID: \"9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5\") " pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:27.531027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530176    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-cni-bin\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.531027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530186    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/54753226-3238-4827-a57c-249e9bb251f8-ovnkube-script-lib\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.531027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530177    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-systemd-units\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.531027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530209    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/df19ed65-1e04-4637-b6d6-8879cd6b750c-dbus\") pod \"global-pull-secret-syncer-csvxz\" (UID: \"df19ed65-1e04-4637-b6d6-8879cd6b750c\") " pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:27.531027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530218    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-systemd-units\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.531027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530237    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/54753226-3238-4827-a57c-249e9bb251f8-env-overrides\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.531027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret\") pod \"global-pull-secret-syncer-csvxz\" (UID: \"df19ed65-1e04-4637-b6d6-8879cd6b750c\") " pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:27.531027 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:27.530340    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 10:53:27.531027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530329    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-slash\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.531027 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:27.530386    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret podName:df19ed65-1e04-4637-b6d6-8879cd6b750c nodeName:}" failed. No retries permitted until 2026-09-17 10:53:28.030370885 +0000 UTC m=+2.128416432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret") pod "global-pull-secret-syncer-csvxz" (UID: "df19ed65-1e04-4637-b6d6-8879cd6b750c") : object "kube-system"/"original-pull-secret" not registered
Sep 17 10:53:27.531027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530403    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-var-lib-openvswitch\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.531027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530408    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-slash\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.531027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4efd824-b4e0-4343-9085-553f8e965648-konnectivity-ca\") pod \"konnectivity-agent-8gz59\" (UID: \"c4efd824-b4e0-4343-9085-553f8e965648\") " pod="kube-system/konnectivity-agent-8gz59"
Sep 17 10:53:27.531027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dcb375fa-358a-4aef-9630-f6bf9c46120b-iptables-alerter-script\") pod \"iptables-alerter-jqhht\" (UID: \"dcb375fa-358a-4aef-9630-f6bf9c46120b\") " pod="openshift-network-operator/iptables-alerter-jqhht"
Sep 17 10:53:27.531027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530480    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-kubelet\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.531027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530490    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-var-lib-openvswitch\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.531027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530504    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-etc-openvswitch\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.531664 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530564    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-host-kubelet\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.531664 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530574    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54753226-3238-4827-a57c-249e9bb251f8-etc-openvswitch\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.531664 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530661    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/54753226-3238-4827-a57c-249e9bb251f8-env-overrides\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.531664 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.530995    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dcb375fa-358a-4aef-9630-f6bf9c46120b-iptables-alerter-script\") pod \"iptables-alerter-jqhht\" (UID: \"dcb375fa-358a-4aef-9630-f6bf9c46120b\") " pod="openshift-network-operator/iptables-alerter-jqhht"
Sep 17 10:53:27.531664 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.531051    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4efd824-b4e0-4343-9085-553f8e965648-konnectivity-ca\") pod \"konnectivity-agent-8gz59\" (UID: \"c4efd824-b4e0-4343-9085-553f8e965648\") " pod="kube-system/konnectivity-agent-8gz59"
Sep 17 10:53:27.532100 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.532076    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/54753226-3238-4827-a57c-249e9bb251f8-ovn-node-metrics-cert\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.532431 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.532415    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4efd824-b4e0-4343-9085-553f8e965648-agent-certs\") pod \"konnectivity-agent-8gz59\" (UID: \"c4efd824-b4e0-4343-9085-553f8e965648\") " pod="kube-system/konnectivity-agent-8gz59"
Sep 17 10:53:27.539153 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:27.539121    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 10:53:27.539153 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:27.539144    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 10:53:27.539324 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:27.539161    2562 projected.go:196] Error preparing data for projected volume kube-api-access-m9jd8 for pod openshift-network-diagnostics/network-check-target-tqgcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 10:53:27.539324 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:27.539224    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8 podName:9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5 nodeName:}" failed. No retries permitted until 2026-09-17 10:53:28.039206701 +0000 UTC m=+2.137252261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m9jd8" (UniqueName: "kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8") pod "network-check-target-tqgcf" (UID: "9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 10:53:27.539530 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.539513    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 10:53:27.540140 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.540119    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4z4m\" (UniqueName: \"kubernetes.io/projected/dcb375fa-358a-4aef-9630-f6bf9c46120b-kube-api-access-s4z4m\") pod \"iptables-alerter-jqhht\" (UID: \"dcb375fa-358a-4aef-9630-f6bf9c46120b\") " pod="openshift-network-operator/iptables-alerter-jqhht"
Sep 17 10:53:27.541150 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.541126    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x898x\" (UniqueName: \"kubernetes.io/projected/54753226-3238-4827-a57c-249e9bb251f8-kube-api-access-x898x\") pod \"ovnkube-node-lzn6w\" (UID: \"54753226-3238-4827-a57c-249e9bb251f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.607472 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:27.607426    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd219867a7ad5b4e9ab4856e1757c643d.slice/crio-bbba6ffb0dd3e84a8dd0b63f25533bce9206c8e3f32d343f8313bf1e9ac37aef WatchSource:0}: Error finding container bbba6ffb0dd3e84a8dd0b63f25533bce9206c8e3f32d343f8313bf1e9ac37aef: Status 404 returned error can't find the container with id bbba6ffb0dd3e84a8dd0b63f25533bce9206c8e3f32d343f8313bf1e9ac37aef
Sep 17 10:53:27.611531 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.611516    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 10:53:27.635201 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.635175    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-flklx"
Sep 17 10:53:27.648401 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:27.648375    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode926f447_9657_48f4_a086_965984dabceb.slice/crio-2fcac5b3d007ccab45b6742b198759c5a682502e2e2a282ea081d94c813b3402 WatchSource:0}: Error finding container 2fcac5b3d007ccab45b6742b198759c5a682502e2e2a282ea081d94c813b3402: Status 404 returned error can't find the container with id 2fcac5b3d007ccab45b6742b198759c5a682502e2e2a282ea081d94c813b3402
Sep 17 10:53:27.649607 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:27.649586    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240331bd5dd4e3d928b56e96a0631dd5.slice/crio-900d2c56ecaa9598f14dbc7771ce2e287d5368c7d54969f308259e65cee8a311 WatchSource:0}: Error finding container 900d2c56ecaa9598f14dbc7771ce2e287d5368c7d54969f308259e65cee8a311: Status 404 returned error can't find the container with id 900d2c56ecaa9598f14dbc7771ce2e287d5368c7d54969f308259e65cee8a311
Sep 17 10:53:27.650646 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.650622    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q5vb8"
Sep 17 10:53:27.661084 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:27.661061    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d6daf4_f47b_4282_8b60_e752eb65e772.slice/crio-967eaee08e7415acc5ee4ebca7522dbc3aea4bc7ec90b70d2a44b3d78a16c25c WatchSource:0}: Error finding container 967eaee08e7415acc5ee4ebca7522dbc3aea4bc7ec90b70d2a44b3d78a16c25c: Status 404 returned error can't find the container with id 967eaee08e7415acc5ee4ebca7522dbc3aea4bc7ec90b70d2a44b3d78a16c25c
Sep 17 10:53:27.673721 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.673696    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-thr8q"
Sep 17 10:53:27.675973 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.675953    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq"
Sep 17 10:53:27.683952 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:27.683933    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e1a707_7a61_4d14_bf87_73bb46d021cd.slice/crio-0f5b75631393bdbb960bbf8ad303426f27bad1f7f10d228053cec58ae5d8198c WatchSource:0}: Error finding container 0f5b75631393bdbb960bbf8ad303426f27bad1f7f10d228053cec58ae5d8198c: Status 404 returned error can't find the container with id 0f5b75631393bdbb960bbf8ad303426f27bad1f7f10d228053cec58ae5d8198c
Sep 17 10:53:27.684988 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:27.684966    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57bf4eda_96bf_428d_8d4e_9ad181d5a8d1.slice/crio-b91f760e8a35d6fc52a06ef92ed86690a0486139a4378484bd1a7bc865cf9de3 WatchSource:0}: Error finding container b91f760e8a35d6fc52a06ef92ed86690a0486139a4378484bd1a7bc865cf9de3: Status 404 returned error can't find the container with id b91f760e8a35d6fc52a06ef92ed86690a0486139a4378484bd1a7bc865cf9de3
Sep 17 10:53:27.696222 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.696196    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hlqjq"
Sep 17 10:53:27.702282 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.702267    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cbwwt"
Sep 17 10:53:27.705485 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:27.705464    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7c0e7e_ab26_4238_b048_869055b359b2.slice/crio-be5956cabc66c7fcb7a97600214b3f7589bfbc9c81c2f8fbd33acab46d0071e1 WatchSource:0}: Error finding container be5956cabc66c7fcb7a97600214b3f7589bfbc9c81c2f8fbd33acab46d0071e1: Status 404 returned error can't find the container with id be5956cabc66c7fcb7a97600214b3f7589bfbc9c81c2f8fbd33acab46d0071e1
Sep 17 10:53:27.706086 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.706066    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jqhht"
Sep 17 10:53:27.710665 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:27.710645    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8778bc91_143b_4854_8393_9e502febb3f5.slice/crio-1186eb288b58dfc0b0e86a10b5a540bc28ad8fdb96f5930f3f4ba799683c0db4 WatchSource:0}: Error finding container 1186eb288b58dfc0b0e86a10b5a540bc28ad8fdb96f5930f3f4ba799683c0db4: Status 404 returned error can't find the container with id 1186eb288b58dfc0b0e86a10b5a540bc28ad8fdb96f5930f3f4ba799683c0db4
Sep 17 10:53:27.715153 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:27.715135    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb375fa_358a_4aef_9630_f6bf9c46120b.slice/crio-4658f05856a13c54554265385c68888a198c71ac8c5b1d2a2e934a6d7362aa4d WatchSource:0}: Error finding container 4658f05856a13c54554265385c68888a198c71ac8c5b1d2a2e934a6d7362aa4d: Status 404 returned error can't find the container with id 4658f05856a13c54554265385c68888a198c71ac8c5b1d2a2e934a6d7362aa4d
Sep 17 10:53:27.750724 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.750697    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:27.755796 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.755760    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8gz59"
Sep 17 10:53:27.760680 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:27.760659    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54753226_3238_4827_a57c_249e9bb251f8.slice/crio-bf30c8384f2b262a01c0f04add856e5d3c2522bc34cf1ed75531e8759a247087 WatchSource:0}: Error finding container bf30c8384f2b262a01c0f04add856e5d3c2522bc34cf1ed75531e8759a247087: Status 404 returned error can't find the container with id bf30c8384f2b262a01c0f04add856e5d3c2522bc34cf1ed75531e8759a247087
Sep 17 10:53:27.765731 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:27.765709    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4efd824_b4e0_4343_9085_553f8e965648.slice/crio-207adda4577f0da2446596d44a1d2234a12ce541f55108052ad93cd08e32f95a WatchSource:0}: Error finding container 207adda4577f0da2446596d44a1d2234a12ce541f55108052ad93cd08e32f95a: Status 404 returned error can't find the container with id 207adda4577f0da2446596d44a1d2234a12ce541f55108052ad93cd08e32f95a
Sep 17 10:53:27.767565 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:27.767540    2562 kuberuntime_manager.go:1459] "Unhandled Error" err="container konnectivity-agent start failed in pod konnectivity-agent-8gz59_kube-system(c4efd824-b4e0-4343-9085-553f8e965648): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 17 10:53:27.767644 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:27.767580    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ErrImagePull: \"pull QPS exceeded\"" pod="kube-system/konnectivity-agent-8gz59" podUID="c4efd824-b4e0-4343-9085-553f8e965648"
Sep 17 10:53:27.933694 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:27.933596    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs\") pod \"network-metrics-daemon-k6wfk\" (UID: \"96dd2d24-1a37-4504-b738-19be03267b6a\") " pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:27.933870 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:27.933709    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 10:53:27.933870 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:27.933784    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs podName:96dd2d24-1a37-4504-b738-19be03267b6a nodeName:}" failed. No retries permitted until 2026-09-17 10:53:28.933768237 +0000 UTC m=+3.031813797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs") pod "network-metrics-daemon-k6wfk" (UID: "96dd2d24-1a37-4504-b738-19be03267b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 10:53:28.034383 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.034342    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret\") pod \"global-pull-secret-syncer-csvxz\" (UID: \"df19ed65-1e04-4637-b6d6-8879cd6b750c\") " pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:28.034574 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:28.034518    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 10:53:28.034628 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:28.034595    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret podName:df19ed65-1e04-4637-b6d6-8879cd6b750c nodeName:}" failed. No retries permitted until 2026-09-17 10:53:29.034575954 +0000 UTC m=+3.132621517 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret") pod "global-pull-secret-syncer-csvxz" (UID: "df19ed65-1e04-4637-b6d6-8879cd6b750c") : object "kube-system"/"original-pull-secret" not registered
Sep 17 10:53:28.135303 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.135264    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jd8\" (UniqueName: \"kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8\") pod \"network-check-target-tqgcf\" (UID: \"9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5\") " pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:28.135495 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:28.135465    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 10:53:28.135495 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:28.135494    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 10:53:28.135603 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:28.135510    2562 projected.go:196] Error preparing data for projected volume kube-api-access-m9jd8 for pod openshift-network-diagnostics/network-check-target-tqgcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 10:53:28.135603 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:28.135576    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8 podName:9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5 nodeName:}" failed. No retries permitted until 2026-09-17 10:53:29.13555481 +0000 UTC m=+3.233600377 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9jd8" (UniqueName: "kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8") pod "network-check-target-tqgcf" (UID: "9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 10:53:28.212603 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.212516    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 10:53:28.390719 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.390652    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 10:48:27 +0000 UTC" deadline="2028-02-13 19:37:09.292601788 +0000 UTC"
Sep 17 10:53:28.390719 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.390717    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12344h43m40.901889916s"
Sep 17 10:53:28.419674 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.419641    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 10:53:28.450504 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.450473    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:28.450683 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:28.450610    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csvxz" podUID="df19ed65-1e04-4637-b6d6-8879cd6b750c"
Sep 17 10:53:28.451176 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.451143    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:28.451293 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:28.451248    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgcf" podUID="9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5"
Sep 17 10:53:28.462553 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.462494    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" event={"ID":"d219867a7ad5b4e9ab4856e1757c643d","Type":"ContainerStarted","Data":"bbba6ffb0dd3e84a8dd0b63f25533bce9206c8e3f32d343f8313bf1e9ac37aef"}
Sep 17 10:53:28.464866 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.464801    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8gz59" event={"ID":"c4efd824-b4e0-4343-9085-553f8e965648","Type":"ContainerStarted","Data":"207adda4577f0da2446596d44a1d2234a12ce541f55108052ad93cd08e32f95a"}
Sep 17 10:53:28.470077 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:28.470051    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb18802deb495c8d9b0ab2c3d8c3043b73eafbad562b04b71a3717870ccdff74\\\": ErrImagePull: pull QPS exceeded\"" pod="kube-system/konnectivity-agent-8gz59" podUID="c4efd824-b4e0-4343-9085-553f8e965648"
Sep 17 10:53:28.474371 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.474340    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jqhht" event={"ID":"dcb375fa-358a-4aef-9630-f6bf9c46120b","Type":"ContainerStarted","Data":"4658f05856a13c54554265385c68888a198c71ac8c5b1d2a2e934a6d7362aa4d"}
Sep 17 10:53:28.486680 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.486648    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cbwwt" event={"ID":"8778bc91-143b-4854-8393-9e502febb3f5","Type":"ContainerStarted","Data":"1186eb288b58dfc0b0e86a10b5a540bc28ad8fdb96f5930f3f4ba799683c0db4"}
Sep 17 10:53:28.492426 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.492400    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq" event={"ID":"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1","Type":"ContainerStarted","Data":"b91f760e8a35d6fc52a06ef92ed86690a0486139a4378484bd1a7bc865cf9de3"}
Sep 17 10:53:28.498989 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.498960    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q5vb8" event={"ID":"16d6daf4-f47b-4282-8b60-e752eb65e772","Type":"ContainerStarted","Data":"967eaee08e7415acc5ee4ebca7522dbc3aea4bc7ec90b70d2a44b3d78a16c25c"}
Sep 17 10:53:28.504695 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.504668    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w" event={"ID":"54753226-3238-4827-a57c-249e9bb251f8","Type":"ContainerStarted","Data":"bf30c8384f2b262a01c0f04add856e5d3c2522bc34cf1ed75531e8759a247087"}
Sep 17 10:53:28.515877 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.515848    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hlqjq" event={"ID":"7e7c0e7e-ab26-4238-b048-869055b359b2","Type":"ContainerStarted","Data":"be5956cabc66c7fcb7a97600214b3f7589bfbc9c81c2f8fbd33acab46d0071e1"}
Sep 17 10:53:28.527063 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.526980    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-thr8q" event={"ID":"03e1a707-7a61-4d14-bf87-73bb46d021cd","Type":"ContainerStarted","Data":"0f5b75631393bdbb960bbf8ad303426f27bad1f7f10d228053cec58ae5d8198c"}
Sep 17 10:53:28.530271 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.530245    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" event={"ID":"240331bd5dd4e3d928b56e96a0631dd5","Type":"ContainerStarted","Data":"900d2c56ecaa9598f14dbc7771ce2e287d5368c7d54969f308259e65cee8a311"}
Sep 17 10:53:28.533978 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.533949    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-flklx" event={"ID":"e926f447-9657-48f4-a086-965984dabceb","Type":"ContainerStarted","Data":"2fcac5b3d007ccab45b6742b198759c5a682502e2e2a282ea081d94c813b3402"}
Sep 17 10:53:28.944848 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:28.942171    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs\") pod \"network-metrics-daemon-k6wfk\" (UID: \"96dd2d24-1a37-4504-b738-19be03267b6a\") " pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:28.944848 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:28.942350    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 10:53:28.944848 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:28.942420    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs podName:96dd2d24-1a37-4504-b738-19be03267b6a nodeName:}" failed. No retries permitted until 2026-09-17 10:53:30.942398813 +0000 UTC m=+5.040444378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs") pod "network-metrics-daemon-k6wfk" (UID: "96dd2d24-1a37-4504-b738-19be03267b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 10:53:29.042974 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:29.042931    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret\") pod \"global-pull-secret-syncer-csvxz\" (UID: \"df19ed65-1e04-4637-b6d6-8879cd6b750c\") " pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:29.043169 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:29.043104    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 10:53:29.043300 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:29.043283    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret podName:df19ed65-1e04-4637-b6d6-8879cd6b750c nodeName:}" failed. No retries permitted until 2026-09-17 10:53:31.043249664 +0000 UTC m=+5.141295225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret") pod "global-pull-secret-syncer-csvxz" (UID: "df19ed65-1e04-4637-b6d6-8879cd6b750c") : object "kube-system"/"original-pull-secret" not registered
Sep 17 10:53:29.144301 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:29.144231    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jd8\" (UniqueName: \"kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8\") pod \"network-check-target-tqgcf\" (UID: \"9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5\") " pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:29.144501 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:29.144484    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 10:53:29.144575 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:29.144507    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 10:53:29.144575 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:29.144522    2562 projected.go:196] Error preparing data for projected volume kube-api-access-m9jd8 for pod openshift-network-diagnostics/network-check-target-tqgcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 10:53:29.144684 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:29.144593    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8 podName:9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5 nodeName:}" failed. No retries permitted until 2026-09-17 10:53:31.144573263 +0000 UTC m=+5.242618815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9jd8" (UniqueName: "kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8") pod "network-check-target-tqgcf" (UID: "9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 10:53:29.391706 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:29.391524    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 10:48:27 +0000 UTC" deadline="2028-04-28 12:54:20.343431201 +0000 UTC"
Sep 17 10:53:29.391706 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:29.391592    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14138h0m50.951843601s"
Sep 17 10:53:29.449687 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:29.449650    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:29.449913 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:29.449812    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6wfk" podUID="96dd2d24-1a37-4504-b738-19be03267b6a"
Sep 17 10:53:30.451744 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:30.451482    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:30.452266 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:30.451856    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csvxz" podUID="df19ed65-1e04-4637-b6d6-8879cd6b750c"
Sep 17 10:53:30.452335 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:30.452267    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:30.452405 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:30.452362    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgcf" podUID="9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5"
Sep 17 10:53:30.963755 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:30.963036    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs\") pod \"network-metrics-daemon-k6wfk\" (UID: \"96dd2d24-1a37-4504-b738-19be03267b6a\") " pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:30.963755 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:30.963209    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 10:53:30.963755 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:30.963279    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs podName:96dd2d24-1a37-4504-b738-19be03267b6a nodeName:}" failed. No retries permitted until 2026-09-17 10:53:34.963258736 +0000 UTC m=+9.061304285 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs") pod "network-metrics-daemon-k6wfk" (UID: "96dd2d24-1a37-4504-b738-19be03267b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 10:53:31.064352 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:31.064312    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret\") pod \"global-pull-secret-syncer-csvxz\" (UID: \"df19ed65-1e04-4637-b6d6-8879cd6b750c\") " pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:31.064550 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:31.064467    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 10:53:31.064550 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:31.064536    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret podName:df19ed65-1e04-4637-b6d6-8879cd6b750c nodeName:}" failed. No retries permitted until 2026-09-17 10:53:35.064517113 +0000 UTC m=+9.162562673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret") pod "global-pull-secret-syncer-csvxz" (UID: "df19ed65-1e04-4637-b6d6-8879cd6b750c") : object "kube-system"/"original-pull-secret" not registered
Sep 17 10:53:31.166194 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:31.165412    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jd8\" (UniqueName: \"kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8\") pod \"network-check-target-tqgcf\" (UID: \"9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5\") " pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:31.166194 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:31.165646    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 10:53:31.166194 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:31.165723    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 10:53:31.166194 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:31.165738    2562 projected.go:196] Error preparing data for projected volume kube-api-access-m9jd8 for pod openshift-network-diagnostics/network-check-target-tqgcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 10:53:31.166194 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:31.165804    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8 podName:9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5 nodeName:}" failed. No retries permitted until 2026-09-17 10:53:35.16578383 +0000 UTC m=+9.263829391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9jd8" (UniqueName: "kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8") pod "network-check-target-tqgcf" (UID: "9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 10:53:31.449785 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:31.449750    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:31.449984 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:31.449917    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6wfk" podUID="96dd2d24-1a37-4504-b738-19be03267b6a"
Sep 17 10:53:32.449754 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:32.449690    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:32.450263 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:32.449834    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgcf" podUID="9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5"
Sep 17 10:53:32.450263 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:32.450241    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:32.450446 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:32.450383    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csvxz" podUID="df19ed65-1e04-4637-b6d6-8879cd6b750c"
Sep 17 10:53:33.450112 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:33.450073    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:33.450502 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:33.450211    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6wfk" podUID="96dd2d24-1a37-4504-b738-19be03267b6a"
Sep 17 10:53:34.450933 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:34.450461    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:34.450933 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:34.450602    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgcf" podUID="9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5"
Sep 17 10:53:34.450933 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:34.450667    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:34.450933 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:34.450849    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csvxz" podUID="df19ed65-1e04-4637-b6d6-8879cd6b750c"
Sep 17 10:53:34.996387 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:34.995833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs\") pod \"network-metrics-daemon-k6wfk\" (UID: \"96dd2d24-1a37-4504-b738-19be03267b6a\") " pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:34.996387 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:34.996009    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 10:53:34.996387 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:34.996073    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs podName:96dd2d24-1a37-4504-b738-19be03267b6a nodeName:}" failed. No retries permitted until 2026-09-17 10:53:42.996053257 +0000 UTC m=+17.094098818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs") pod "network-metrics-daemon-k6wfk" (UID: "96dd2d24-1a37-4504-b738-19be03267b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 10:53:35.097317 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:35.097043    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret\") pod \"global-pull-secret-syncer-csvxz\" (UID: \"df19ed65-1e04-4637-b6d6-8879cd6b750c\") " pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:35.097317 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:35.097172    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 10:53:35.097317 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:35.097244    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret podName:df19ed65-1e04-4637-b6d6-8879cd6b750c nodeName:}" failed. No retries permitted until 2026-09-17 10:53:43.097224058 +0000 UTC m=+17.195269626 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret") pod "global-pull-secret-syncer-csvxz" (UID: "df19ed65-1e04-4637-b6d6-8879cd6b750c") : object "kube-system"/"original-pull-secret" not registered
Sep 17 10:53:35.197976 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:35.197883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jd8\" (UniqueName: \"kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8\") pod \"network-check-target-tqgcf\" (UID: \"9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5\") " pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:35.198130 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:35.198044    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 10:53:35.198130 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:35.198073    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 10:53:35.198130 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:35.198086    2562 projected.go:196] Error preparing data for projected volume kube-api-access-m9jd8 for pod openshift-network-diagnostics/network-check-target-tqgcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 10:53:35.198291 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:35.198155    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8 podName:9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5 nodeName:}" failed. No retries permitted until 2026-09-17 10:53:43.198134057 +0000 UTC m=+17.296179614 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9jd8" (UniqueName: "kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8") pod "network-check-target-tqgcf" (UID: "9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 10:53:35.449825 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:35.449737    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:35.449974 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:35.449870    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6wfk" podUID="96dd2d24-1a37-4504-b738-19be03267b6a"
Sep 17 10:53:36.451274 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:36.450836    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:36.451274 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:36.450948    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csvxz" podUID="df19ed65-1e04-4637-b6d6-8879cd6b750c"
Sep 17 10:53:36.451274 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:36.451123    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:36.451274 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:36.451216    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgcf" podUID="9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5"
Sep 17 10:53:37.449603 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:37.449574    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:37.449785 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:37.449697    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6wfk" podUID="96dd2d24-1a37-4504-b738-19be03267b6a"
Sep 17 10:53:38.449311 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:38.449262    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:38.449811 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:38.449266    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:38.449811 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:38.449399    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgcf" podUID="9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5"
Sep 17 10:53:38.449811 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:38.449464    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csvxz" podUID="df19ed65-1e04-4637-b6d6-8879cd6b750c"
Sep 17 10:53:39.449468 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:39.449429    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:39.449902 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:39.449580    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6wfk" podUID="96dd2d24-1a37-4504-b738-19be03267b6a"
Sep 17 10:53:40.449157 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:40.449092    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:40.449343 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:40.449242    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csvxz" podUID="df19ed65-1e04-4637-b6d6-8879cd6b750c"
Sep 17 10:53:40.449343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:40.449304    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:40.449464 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:40.449421    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgcf" podUID="9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5"
Sep 17 10:53:41.449832 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:41.449783    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:41.450222 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:41.450014    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6wfk" podUID="96dd2d24-1a37-4504-b738-19be03267b6a"
Sep 17 10:53:42.449687 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:42.449599    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:42.449888 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:42.449604    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:42.449888 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:42.449733    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgcf" podUID="9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5"
Sep 17 10:53:42.449888 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:42.449827    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csvxz" podUID="df19ed65-1e04-4637-b6d6-8879cd6b750c"
Sep 17 10:53:43.060363 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:43.060328    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs\") pod \"network-metrics-daemon-k6wfk\" (UID: \"96dd2d24-1a37-4504-b738-19be03267b6a\") " pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:43.060569 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:43.060465    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 10:53:43.060569 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:43.060536    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs podName:96dd2d24-1a37-4504-b738-19be03267b6a nodeName:}" failed. No retries permitted until 2026-09-17 10:53:59.060515152 +0000 UTC m=+33.158560713 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs") pod "network-metrics-daemon-k6wfk" (UID: "96dd2d24-1a37-4504-b738-19be03267b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 10:53:43.161253 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:43.161219    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret\") pod \"global-pull-secret-syncer-csvxz\" (UID: \"df19ed65-1e04-4637-b6d6-8879cd6b750c\") " pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:43.161428 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:43.161366    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 10:53:43.161478 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:43.161441    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret podName:df19ed65-1e04-4637-b6d6-8879cd6b750c nodeName:}" failed. No retries permitted until 2026-09-17 10:53:59.161423535 +0000 UTC m=+33.259469078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret") pod "global-pull-secret-syncer-csvxz" (UID: "df19ed65-1e04-4637-b6d6-8879cd6b750c") : object "kube-system"/"original-pull-secret" not registered
Sep 17 10:53:43.262548 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:43.262511    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jd8\" (UniqueName: \"kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8\") pod \"network-check-target-tqgcf\" (UID: \"9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5\") " pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:43.262731 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:43.262674    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 10:53:43.262731 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:43.262696    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 10:53:43.262731 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:43.262705    2562 projected.go:196] Error preparing data for projected volume kube-api-access-m9jd8 for pod openshift-network-diagnostics/network-check-target-tqgcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 10:53:43.262901 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:43.262766    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8 podName:9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5 nodeName:}" failed. No retries permitted until 2026-09-17 10:53:59.262746473 +0000 UTC m=+33.360792030 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9jd8" (UniqueName: "kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8") pod "network-check-target-tqgcf" (UID: "9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 10:53:43.449857 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:43.449770    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:43.450006 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:43.449913    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6wfk" podUID="96dd2d24-1a37-4504-b738-19be03267b6a"
Sep 17 10:53:44.449450 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:44.449410    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:44.449623 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:44.449526    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgcf" podUID="9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5"
Sep 17 10:53:44.449623 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:44.449565    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:44.449729 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:44.449645    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csvxz" podUID="df19ed65-1e04-4637-b6d6-8879cd6b750c"
Sep 17 10:53:45.449252 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:45.449214    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:45.449805 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:45.449352    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6wfk" podUID="96dd2d24-1a37-4504-b738-19be03267b6a"
Sep 17 10:53:46.452079 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:46.451997    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:46.452698 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:46.452105    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgcf" podUID="9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5"
Sep 17 10:53:46.452698 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:46.452401    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:46.452698 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:46.452509    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csvxz" podUID="df19ed65-1e04-4637-b6d6-8879cd6b750c"
Sep 17 10:53:46.617008 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:46.616700    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w" event={"ID":"54753226-3238-4827-a57c-249e9bb251f8","Type":"ContainerStarted","Data":"2069dfb0ff464e32afeac42d0e2d300e29f908892b3082cc092988fe2fb3dacd"}
Sep 17 10:53:46.617008 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:46.616980    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w" event={"ID":"54753226-3238-4827-a57c-249e9bb251f8","Type":"ContainerStarted","Data":"739024f00ea6d28052ede6d2876b7acf56ff466201f72e49dec62ecdd7b30204"}
Sep 17 10:53:46.624552 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:46.624509    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-flklx" event={"ID":"e926f447-9657-48f4-a086-965984dabceb","Type":"ContainerStarted","Data":"79c02baaf741e3aa047ebd9050a42348ecf255a22036b04e568842097c86dbc2"}
Sep 17 10:53:46.636169 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:46.636130    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" event={"ID":"d219867a7ad5b4e9ab4856e1757c643d","Type":"ContainerStarted","Data":"9a1a7cd96d5a3c7e3e8a9ecec0e797fc9a75da63afd5a07221d7e454b6025c0c"}
Sep 17 10:53:46.639292 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:46.639264    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cbwwt" event={"ID":"8778bc91-143b-4854-8393-9e502febb3f5","Type":"ContainerStarted","Data":"d5825ff193e5ad9d01e4cffada78f90f8e1c0fc91112d76b51d5caf33e604c46"}
Sep 17 10:53:46.644591 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:46.643789    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-flklx" podStartSLOduration=1.081467474 podStartE2EDuration="19.643776731s" totalImagesPullingTime="18.562309257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:53:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:53:46.64320451 +0000 UTC m=+20.741250077" watchObservedRunningTime="2026-09-17 10:53:46.643776731 +0000 UTC m=+20.741822296"
Sep 17 10:53:46.669023 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:46.668969    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cbwwt" podStartSLOduration=0.926229105 podStartE2EDuration="19.668950343s" totalImagesPullingTime="18.742721238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:53:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:53:46.668943068 +0000 UTC m=+20.766988662" watchObservedRunningTime="2026-09-17 10:53:46.668950343 +0000 UTC m=+20.766995911"
Sep 17 10:53:46.691316 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:46.691261    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" podStartSLOduration=19.691240602 podStartE2EDuration="19.691240602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:53:46.69031036 +0000 UTC m=+20.788355928" watchObservedRunningTime="2026-09-17 10:53:46.691240602 +0000 UTC m=+20.789286168"
Sep 17 10:53:47.449230 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:47.449071    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:47.449339 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:47.449320    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6wfk" podUID="96dd2d24-1a37-4504-b738-19be03267b6a"
Sep 17 10:53:47.643752 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:47.643714    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8gz59" event={"ID":"c4efd824-b4e0-4343-9085-553f8e965648","Type":"ContainerStarted","Data":"a28db9739ff862eba1081d6000a4ecc83a524929cf5fe9ec0feff1383124640d"}
Sep 17 10:53:47.645424 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:47.645382    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq" event={"ID":"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1","Type":"ContainerStarted","Data":"21a4e3aaedc461db6504a770dfc53913025d38bb03a11b176f0fcfea7d98f840"}
Sep 17 10:53:47.646975 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:47.646948    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q5vb8" event={"ID":"16d6daf4-f47b-4282-8b60-e752eb65e772","Type":"ContainerStarted","Data":"0f4005933fc07db2ca40c66060ba78e23690ab084d8ee3761ecacf6b9befa209"}
Sep 17 10:53:47.650396 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:47.650335    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w" event={"ID":"54753226-3238-4827-a57c-249e9bb251f8","Type":"ContainerStarted","Data":"3fcd79dbe93af6003e6759ada91b9886a4954b85b90af7e28c1cdbe44f88078a"}
Sep 17 10:53:47.650482 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:47.650401    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w" event={"ID":"54753226-3238-4827-a57c-249e9bb251f8","Type":"ContainerStarted","Data":"ed5b13ca6bffc8a590a962c7740550cdb63291742792c4607fb7b510fc6ea1a2"}
Sep 17 10:53:47.650482 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:47.650444    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w" event={"ID":"54753226-3238-4827-a57c-249e9bb251f8","Type":"ContainerStarted","Data":"e8c4cbe1c52232d9843802eae3cc06d66d942c4e8dc464fe21292359d0fde568"}
Sep 17 10:53:47.650625 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:47.650486    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w" event={"ID":"54753226-3238-4827-a57c-249e9bb251f8","Type":"ContainerStarted","Data":"98612c8dbb4b7d8bee043eb7913410cc96cf5c579df92dbff7efdd09c74c951e"}
Sep 17 10:53:47.651997 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:47.651973    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hlqjq" event={"ID":"7e7c0e7e-ab26-4238-b048-869055b359b2","Type":"ContainerStarted","Data":"dd1e3db7aebd2ced99a2aa09278778f9f170fe4b02407dd44b20c674bb404a12"}
Sep 17 10:53:47.653532 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:47.653508    2562 generic.go:356] "Generic (PLEG): container finished" podID="03e1a707-7a61-4d14-bf87-73bb46d021cd" containerID="a994f61468482013690636e2e81ab0a7c6946e283ecdbfea0da316c20a91e353" exitCode=0
Sep 17 10:53:47.653602 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:47.653577    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-thr8q" event={"ID":"03e1a707-7a61-4d14-bf87-73bb46d021cd","Type":"ContainerDied","Data":"a994f61468482013690636e2e81ab0a7c6946e283ecdbfea0da316c20a91e353"}
Sep 17 10:53:47.655480 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:47.655269    2562 generic.go:356] "Generic (PLEG): container finished" podID="240331bd5dd4e3d928b56e96a0631dd5" containerID="1920d02ed53842186fcc174409666c6bc6c4d5178f98991f6ba2b821010953ae" exitCode=0
Sep 17 10:53:47.655480 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:47.655307    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" event={"ID":"240331bd5dd4e3d928b56e96a0631dd5","Type":"ContainerDied","Data":"1920d02ed53842186fcc174409666c6bc6c4d5178f98991f6ba2b821010953ae"}
Sep 17 10:53:47.671155 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:47.671101    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8gz59" podStartSLOduration=0.988117357 podStartE2EDuration="20.671082225s" totalImagesPullingTime="19.682964868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:53:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-17 10:53:47.667650725 +0000 UTC m=+21.765696292" watchObservedRunningTime="2026-09-17 10:53:47.671082225 +0000 UTC m=+21.769127788"
Sep 17 10:53:47.734123 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:47.734069    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hlqjq" podStartSLOduration=2.229890293 podStartE2EDuration="20.734047552s" totalImagesPullingTime="18.504157259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:53:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:53:47.733497648 +0000 UTC m=+21.831543214" watchObservedRunningTime="2026-09-17 10:53:47.734047552 +0000 UTC m=+21.832093119"
Sep 17 10:53:47.752304 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:47.752257    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q5vb8" podStartSLOduration=2.225832808 podStartE2EDuration="20.752235258s" totalImagesPullingTime="18.52640245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:53:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:53:47.751937326 +0000 UTC m=+21.849982892" watchObservedRunningTime="2026-09-17 10:53:47.752235258 +0000 UTC m=+21.850280823"
Sep 17 10:53:47.999526 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:47.999502    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 10:53:48.389600 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:48.389450    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T10:53:47.99952237Z","UUID":"d8bfe178-f90c-4352-821d-8288aef7fc68","Handler":null,"Name":"","Endpoint":""}
Sep 17 10:53:48.391195 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:48.391063    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 10:53:48.391298 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:48.391204    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 10:53:48.449651 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:48.449622    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:48.449810 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:48.449751    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csvxz" podUID="df19ed65-1e04-4637-b6d6-8879cd6b750c"
Sep 17 10:53:48.449810 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:48.449799    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:48.449918 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:48.449901    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgcf" podUID="9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5"
Sep 17 10:53:48.659865 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:48.659375    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jqhht" event={"ID":"dcb375fa-358a-4aef-9630-f6bf9c46120b","Type":"ContainerStarted","Data":"aada0dfba624c2fa2e239e2005fe1526c84cd45320fd5c917aa59b0e88594902"}
Sep 17 10:53:48.661781 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:48.661746    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq" event={"ID":"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1","Type":"ContainerStarted","Data":"172ee93a7203905ee5557128b944ce10fb7daf53e8d8e07633c3b2424fd8fca6"}
Sep 17 10:53:48.664328 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:48.664257    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" event={"ID":"240331bd5dd4e3d928b56e96a0631dd5","Type":"ContainerStarted","Data":"0b5623f457f4d1f2f40b11010154530c488fd5e55cc728553a33ac012a3366d3"}
Sep 17 10:53:48.676842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:48.676782    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jqhht" podStartSLOduration=3.180902854 podStartE2EDuration="21.676762841s" totalImagesPullingTime="18.495859987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:53:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:53:48.675980851 +0000 UTC m=+22.774026438" watchObservedRunningTime="2026-09-17 10:53:48.676762841 +0000 UTC m=+22.774808410"
Sep 17 10:53:49.449552 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:49.449465    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:49.449702 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:49.449645    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6wfk" podUID="96dd2d24-1a37-4504-b738-19be03267b6a"
Sep 17 10:53:49.670278 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:49.670236    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq" event={"ID":"57bf4eda-96bf-428d-8d4e-9ad181d5a8d1","Type":"ContainerStarted","Data":"d8196bbe4b57ef6072d854255cf60a218fe142a5934f03b568a6aae3d7a124e8"}
Sep 17 10:53:49.674133 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:49.674102    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w" event={"ID":"54753226-3238-4827-a57c-249e9bb251f8","Type":"ContainerStarted","Data":"8debc1637d518e831d8ded20191255b2344960c81756abd72ed0e70963559dee"}
Sep 17 10:53:49.704951 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:49.704861    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" podStartSLOduration=22.704843683 podStartE2EDuration="22.704843683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:53:48.696935581 +0000 UTC m=+22.794981150" watchObservedRunningTime="2026-09-17 10:53:49.704843683 +0000 UTC m=+23.802889271"
Sep 17 10:53:50.449543 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:50.449503    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:50.449838 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:50.449517    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:50.449838 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:50.449625    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csvxz" podUID="df19ed65-1e04-4637-b6d6-8879cd6b750c"
Sep 17 10:53:50.449838 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:50.449703    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgcf" podUID="9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5"
Sep 17 10:53:50.929780 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:50.929686    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8gz59"
Sep 17 10:53:50.930444 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:50.930426    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8gz59"
Sep 17 10:53:50.949172 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:50.949123    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz8tq" podStartSLOduration=3.235172144 podStartE2EDuration="23.949100584s" totalImagesPullingTime="20.71392844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:53:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:53:49.704981562 +0000 UTC m=+23.803027127" watchObservedRunningTime="2026-09-17 10:53:50.949100584 +0000 UTC m=+25.047146152"
Sep 17 10:53:51.449851 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:51.449807    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:51.450006 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:51.449961    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6wfk" podUID="96dd2d24-1a37-4504-b738-19be03267b6a"
Sep 17 10:53:51.679547 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:51.679472    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8gz59"
Sep 17 10:53:51.679943 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:51.679922    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8gz59"
Sep 17 10:53:52.449922 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:52.449899    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:52.450301 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:52.449899    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:52.450301 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:52.450001    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csvxz" podUID="df19ed65-1e04-4637-b6d6-8879cd6b750c"
Sep 17 10:53:52.450301 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:52.450085    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgcf" podUID="9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5"
Sep 17 10:53:52.685615 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:52.685333    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w" event={"ID":"54753226-3238-4827-a57c-249e9bb251f8","Type":"ContainerStarted","Data":"b5be3f5771a4d14a89f9d219753bffe4e45ca51d448bd458093fd65c151b5980"}
Sep 17 10:53:52.716977 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:52.716929    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w" podStartSLOduration=7.255579887 podStartE2EDuration="25.716914197s" totalImagesPullingTime="18.46133431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:53:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:53:52.715351347 +0000 UTC m=+26.813396913" watchObservedRunningTime="2026-09-17 10:53:52.716914197 +0000 UTC m=+26.814959763"
Sep 17 10:53:53.449590 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:53.449561    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:53.449812 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:53.449668    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6wfk" podUID="96dd2d24-1a37-4504-b738-19be03267b6a"
Sep 17 10:53:53.689744 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:53.689707    2562 generic.go:356] "Generic (PLEG): container finished" podID="03e1a707-7a61-4d14-bf87-73bb46d021cd" containerID="d5e9afe8753300864084483c7feefe04422a8db245db9a0c587a876925580052" exitCode=0
Sep 17 10:53:53.690332 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:53.689786    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-thr8q" event={"ID":"03e1a707-7a61-4d14-bf87-73bb46d021cd","Type":"ContainerDied","Data":"d5e9afe8753300864084483c7feefe04422a8db245db9a0c587a876925580052"}
Sep 17 10:53:53.690332 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:53.690076    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:53.690332 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:53.690283    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:53.690482 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:53.690340    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:53.721279 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:53.721116    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:53.727124 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:53.727001    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:53:54.449304 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:54.449278    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:54.449567 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:54.449420    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csvxz" podUID="df19ed65-1e04-4637-b6d6-8879cd6b750c"
Sep 17 10:53:54.449567 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:54.449466    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:54.449567 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:54.449553    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgcf" podUID="9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5"
Sep 17 10:53:54.538709 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:54.538672    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k6wfk"]
Sep 17 10:53:54.538867 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:54.538788    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:54.538954 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:54.538930    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6wfk" podUID="96dd2d24-1a37-4504-b738-19be03267b6a"
Sep 17 10:53:54.543114 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:54.543087    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tqgcf"]
Sep 17 10:53:54.543785 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:54.543762    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-csvxz"]
Sep 17 10:53:54.694695 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:54.694609    2562 generic.go:356] "Generic (PLEG): container finished" podID="03e1a707-7a61-4d14-bf87-73bb46d021cd" containerID="1ced9252dbf91d02b07bcf8e42fdcd80aaf36658587b672e31981660ad27863a" exitCode=0
Sep 17 10:53:54.695385 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:54.694719    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:54.695385 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:54.694715    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-thr8q" event={"ID":"03e1a707-7a61-4d14-bf87-73bb46d021cd","Type":"ContainerDied","Data":"1ced9252dbf91d02b07bcf8e42fdcd80aaf36658587b672e31981660ad27863a"}
Sep 17 10:53:54.695385 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:54.694856    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:54.695385 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:54.694954    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csvxz" podUID="df19ed65-1e04-4637-b6d6-8879cd6b750c"
Sep 17 10:53:54.696889 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:54.696857    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgcf" podUID="9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5"
Sep 17 10:53:56.451434 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:56.451183    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:56.451830 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:56.451245    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:56.451830 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:56.451527    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csvxz" podUID="df19ed65-1e04-4637-b6d6-8879cd6b750c"
Sep 17 10:53:56.451830 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:56.451266    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:56.451830 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:56.451597    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6wfk" podUID="96dd2d24-1a37-4504-b738-19be03267b6a"
Sep 17 10:53:56.451830 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:56.451657    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgcf" podUID="9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5"
Sep 17 10:53:56.701078 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:56.701046    2562 generic.go:356] "Generic (PLEG): container finished" podID="03e1a707-7a61-4d14-bf87-73bb46d021cd" containerID="7b37abcded2d5490c8333e0d6f82c09fb694b3777e53473160a108a147eb463a" exitCode=0
Sep 17 10:53:56.701265 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:56.701088    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-thr8q" event={"ID":"03e1a707-7a61-4d14-bf87-73bb46d021cd","Type":"ContainerDied","Data":"7b37abcded2d5490c8333e0d6f82c09fb694b3777e53473160a108a147eb463a"}
Sep 17 10:53:58.450113 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:58.450076    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:58.450776 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:58.450080    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:58.450776 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:58.450217    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csvxz" podUID="df19ed65-1e04-4637-b6d6-8879cd6b750c"
Sep 17 10:53:58.450776 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:58.450089    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:58.450776 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:58.450295    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6wfk" podUID="96dd2d24-1a37-4504-b738-19be03267b6a"
Sep 17 10:53:58.450776 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:58.450351    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqgcf" podUID="9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5"
Sep 17 10:53:59.078883 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.078842    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs\") pod \"network-metrics-daemon-k6wfk\" (UID: \"96dd2d24-1a37-4504-b738-19be03267b6a\") " pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:53:59.079024 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.078984    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 10:53:59.079077 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.079051    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs podName:96dd2d24-1a37-4504-b738-19be03267b6a nodeName:}" failed. No retries permitted until 2026-09-17 10:54:31.079035767 +0000 UTC m=+65.177081311 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs") pod "network-metrics-daemon-k6wfk" (UID: "96dd2d24-1a37-4504-b738-19be03267b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 10:53:59.107948 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.107923    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeReady"
Sep 17 10:53:59.108074 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.108054    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 10:53:59.151212 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.151181    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-dpdf4"]
Sep 17 10:53:59.175702 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.175666    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"]
Sep 17 10:53:59.176122 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.176075    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.179841 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.179627    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret\") pod \"global-pull-secret-syncer-csvxz\" (UID: \"df19ed65-1e04-4637-b6d6-8879cd6b750c\") " pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:53:59.179841 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.179776    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 10:53:59.180029 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.179859    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret podName:df19ed65-1e04-4637-b6d6-8879cd6b750c nodeName:}" failed. No retries permitted until 2026-09-17 10:54:31.179833647 +0000 UTC m=+65.277879218 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret") pod "global-pull-secret-syncer-csvxz" (UID: "df19ed65-1e04-4637-b6d6-8879cd6b750c") : object "kube-system"/"original-pull-secret" not registered
Sep 17 10:53:59.184027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.184002    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-c9k86\""
Sep 17 10:53:59.184354 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.184189    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 10:53:59.184354 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.184219    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 10:53:59.184354 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.184123    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 10:53:59.186840 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.186623    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 10:53:59.191746 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.191724    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 10:53:59.192882 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.192863    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7fc56fb846-t46f7"]
Sep 17 10:53:59.193039 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.193022    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"
Sep 17 10:53:59.196649 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.196628    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 10:53:59.196866 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.196843    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7vsv8\""
Sep 17 10:53:59.196975 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.196894    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 10:53:59.197138 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.197122    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 10:53:59.197291 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.197271    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 10:53:59.217206 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.216935    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8fd5d4c56-t84nw"]
Sep 17 10:53:59.217206 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.217112    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:53:59.220631 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.220606    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 10:53:59.220975 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.220940    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 10:53:59.221199 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.221178    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 10:53:59.221321 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.221298    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xsc6h\""
Sep 17 10:53:59.221614 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.221301    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 10:53:59.221951 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.221936    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 10:53:59.222168 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.222153    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 10:53:59.232448 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.232426    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"]
Sep 17 10:53:59.232579 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.232472    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r"]
Sep 17 10:53:59.232579 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.232545    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.236618 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.236497    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 10:53:59.236618 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.236575    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 10:53:59.236618 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.236590    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 10:53:59.237332 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.236738    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n4lnv\""
Sep 17 10:53:59.242452 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.242435    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 10:53:59.255173 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.255148    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-tt7px"]
Sep 17 10:53:59.255290 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.255278    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r"
Sep 17 10:53:59.259301 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.259283    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 10:53:59.259548 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.259533    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 10:53:59.259548 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.259542    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-nd2tn\""
Sep 17 10:53:59.259657 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.259565    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 10:53:59.259657 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.259589    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 10:53:59.273236 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.273206    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq"]
Sep 17 10:53:59.273367 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.273346    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-tt7px"
Sep 17 10:53:59.276348 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.276326    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 10:53:59.276532 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.276360    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wvsmc\""
Sep 17 10:53:59.276532 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.276520    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 10:53:59.279999 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.279979    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a72b5be-ac50-4f83-89ab-f0664cf51f1f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-dpdf4\" (UID: \"3a72b5be-ac50-4f83-89ab-f0664cf51f1f\") " pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.280067 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.280030    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89xtc\" (UniqueName: \"kubernetes.io/projected/3738db4a-c2eb-42b9-ad15-280d36c6b75e-kube-api-access-89xtc\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:53:59.280105 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.280067    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jd8\" (UniqueName: \"kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8\") pod \"network-check-target-tqgcf\" (UID: \"9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5\") " pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:59.280105 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.280098    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:53:59.280166 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.280135    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-stats-auth\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:53:59.280198 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.280164    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l72b\" (UniqueName: \"kubernetes.io/projected/3a72b5be-ac50-4f83-89ab-f0664cf51f1f-kube-api-access-7l72b\") pod \"insights-operator-5844b856db-dpdf4\" (UID: \"3a72b5be-ac50-4f83-89ab-f0664cf51f1f\") " pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.280198 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.280191    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3a72b5be-ac50-4f83-89ab-f0664cf51f1f-snapshots\") pod \"insights-operator-5844b856db-dpdf4\" (UID: \"3a72b5be-ac50-4f83-89ab-f0664cf51f1f\") " pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.280260 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.280234    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2v8pp\" (UID: \"f7c60693-7739-4de7-9a76-13ce9a464eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"
Sep 17 10:53:59.280294 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.280277    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-default-certificate\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:53:59.280324 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.280303    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:53:59.280352 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.280340    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a72b5be-ac50-4f83-89ab-f0664cf51f1f-serving-cert\") pod \"insights-operator-5844b856db-dpdf4\" (UID: \"3a72b5be-ac50-4f83-89ab-f0664cf51f1f\") " pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.280384 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.280364    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6v4g\" (UniqueName: \"kubernetes.io/projected/f7c60693-7739-4de7-9a76-13ce9a464eb8-kube-api-access-l6v4g\") pod \"cluster-monitoring-operator-6cd954b897-2v8pp\" (UID: \"f7c60693-7739-4de7-9a76-13ce9a464eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"
Sep 17 10:53:59.280416 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.280392    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a72b5be-ac50-4f83-89ab-f0664cf51f1f-service-ca-bundle\") pod \"insights-operator-5844b856db-dpdf4\" (UID: \"3a72b5be-ac50-4f83-89ab-f0664cf51f1f\") " pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.280448 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.280413    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f7c60693-7739-4de7-9a76-13ce9a464eb8-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-2v8pp\" (UID: \"f7c60693-7739-4de7-9a76-13ce9a464eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"
Sep 17 10:53:59.280448 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.280438    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a72b5be-ac50-4f83-89ab-f0664cf51f1f-tmp\") pod \"insights-operator-5844b856db-dpdf4\" (UID: \"3a72b5be-ac50-4f83-89ab-f0664cf51f1f\") " pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.285725 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.285695    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9jd8\" (UniqueName: \"kubernetes.io/projected/9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5-kube-api-access-m9jd8\") pod \"network-check-target-tqgcf\" (UID: \"9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5\") " pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:53:59.297552 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.297529    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-frdv7"]
Sep 17 10:53:59.297699 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.297673    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq"
Sep 17 10:53:59.301048 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.301018    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-t2kxj\""
Sep 17 10:53:59.301345 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.301326    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 10:53:59.301457 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.301363    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 10:53:59.301457 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.301398    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 10:53:59.322763 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.322735    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t"]
Sep 17 10:53:59.322931 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.322893    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-frdv7"
Sep 17 10:53:59.326052 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.326029    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 10:53:59.326191 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.326139    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-j6rx2\""
Sep 17 10:53:59.326191 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.326160    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 10:53:59.337664 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.337642    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-dpdf4"]
Sep 17 10:53:59.337784 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.337680    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-dz7s6"]
Sep 17 10:53:59.337784 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.337774    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t"
Sep 17 10:53:59.340630 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.340500    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 10:53:59.340630 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.340513    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 10:53:59.340630 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.340547    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 10:53:59.340630 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.340539    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 10:53:59.340928 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.340689    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vhqpk\""
Sep 17 10:53:59.357835 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.357793    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jcmwx"]
Sep 17 10:53:59.357946 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.357917    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-dz7s6"
Sep 17 10:53:59.360691 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.360666    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 10:53:59.360827 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.360724    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-n6gmc\""
Sep 17 10:53:59.360986 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.360971    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 10:53:59.361256 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.361237    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 10:53:59.362086 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.362059    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 10:53:59.365619 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.365596    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 10:53:59.377070 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.377044    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8tngn"]
Sep 17 10:53:59.377217 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.377192    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:53:59.379824 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.379685    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 10:53:59.379824 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.379808    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 10:53:59.380094 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.380078    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lfdp8\""
Sep 17 10:53:59.381283 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.381261    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-bound-sa-token\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.381391 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.381296    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.381391 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.381321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-certificates\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.381391 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.381367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrgn6\" (UniqueName: \"kubernetes.io/projected/b1ded66f-7439-4acc-a8e6-9a3b206d9af7-kube-api-access-hrgn6\") pod \"network-check-source-6f8b768455-tt7px\" (UID: \"b1ded66f-7439-4acc-a8e6-9a3b206d9af7\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-tt7px"
Sep 17 10:53:59.381523 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.381423    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l72b\" (UniqueName: \"kubernetes.io/projected/3a72b5be-ac50-4f83-89ab-f0664cf51f1f-kube-api-access-7l72b\") pod \"insights-operator-5844b856db-dpdf4\" (UID: \"3a72b5be-ac50-4f83-89ab-f0664cf51f1f\") " pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.381523 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.381449    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-stats-auth\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:53:59.381611 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.381597    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqkfc\" (UniqueName: \"kubernetes.io/projected/ad3b878d-f419-4e39-88d1-a53ee931a22e-kube-api-access-kqkfc\") pod \"service-ca-operator-cf7c776b7-mrd5r\" (UID: \"ad3b878d-f419-4e39-88d1-a53ee931a22e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r"
Sep 17 10:53:59.381664 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.381639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2v8pp\" (UID: \"f7c60693-7739-4de7-9a76-13ce9a464eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"
Sep 17 10:53:59.381710 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.381671    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad3b878d-f419-4e39-88d1-a53ee931a22e-config\") pod \"service-ca-operator-cf7c776b7-mrd5r\" (UID: \"ad3b878d-f419-4e39-88d1-a53ee931a22e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r"
Sep 17 10:53:59.381710 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.381702    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6v4g\" (UniqueName: \"kubernetes.io/projected/f7c60693-7739-4de7-9a76-13ce9a464eb8-kube-api-access-l6v4g\") pod \"cluster-monitoring-operator-6cd954b897-2v8pp\" (UID: \"f7c60693-7739-4de7-9a76-13ce9a464eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"
Sep 17 10:53:59.381806 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.381732    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a72b5be-ac50-4f83-89ab-f0664cf51f1f-service-ca-bundle\") pod \"insights-operator-5844b856db-dpdf4\" (UID: \"3a72b5be-ac50-4f83-89ab-f0664cf51f1f\") " pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.381806 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.381766    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-image-registry-private-configuration\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.381806 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.381786    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 10:53:59.381970 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.381839    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a72b5be-ac50-4f83-89ab-f0664cf51f1f-tmp\") pod \"insights-operator-5844b856db-dpdf4\" (UID: \"3a72b5be-ac50-4f83-89ab-f0664cf51f1f\") " pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.381970 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.381871    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls podName:f7c60693-7739-4de7-9a76-13ce9a464eb8 nodeName:}" failed. No retries permitted until 2026-09-17 10:53:59.881851097 +0000 UTC m=+33.979896647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-2v8pp" (UID: "f7c60693-7739-4de7-9a76-13ce9a464eb8") : secret "cluster-monitoring-operator-tls" not found
Sep 17 10:53:59.381970 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.381912    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-installation-pull-secrets\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.381970 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.381940    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad3b878d-f419-4e39-88d1-a53ee931a22e-serving-cert\") pod \"service-ca-operator-cf7c776b7-mrd5r\" (UID: \"ad3b878d-f419-4e39-88d1-a53ee931a22e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r"
Sep 17 10:53:59.382172 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.381973    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nq9rq\" (UID: \"d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq"
Sep 17 10:53:59.382172 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.381997    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xct2n\" (UniqueName: \"kubernetes.io/projected/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-kube-api-access-xct2n\") pod \"cluster-samples-operator-86499c89c4-nq9rq\" (UID: \"d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq"
Sep 17 10:53:59.382172 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.382021    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-ca-trust-extracted\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.382172 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.382050    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:53:59.382172 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.382080    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n44vg\" (UniqueName: \"kubernetes.io/projected/c5717a5d-758e-4255-a058-84bfefce6fbc-kube-api-access-n44vg\") pod \"volume-data-source-validator-86695b7cc7-frdv7\" (UID: \"c5717a5d-758e-4255-a058-84bfefce6fbc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-frdv7"
Sep 17 10:53:59.382172 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.382110    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89xtc\" (UniqueName: \"kubernetes.io/projected/3738db4a-c2eb-42b9-ad15-280d36c6b75e-kube-api-access-89xtc\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:53:59.382172 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.382165    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-default-certificate\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:53:59.382545 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.382268    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 10:53:59.382545 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.382341    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs podName:3738db4a-c2eb-42b9-ad15-280d36c6b75e nodeName:}" failed. No retries permitted until 2026-09-17 10:53:59.882322018 +0000 UTC m=+33.980367570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs") pod "router-default-7fc56fb846-t46f7" (UID: "3738db4a-c2eb-42b9-ad15-280d36c6b75e") : secret "router-metrics-certs-default" not found
Sep 17 10:53:59.382545 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.382389    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3a72b5be-ac50-4f83-89ab-f0664cf51f1f-snapshots\") pod \"insights-operator-5844b856db-dpdf4\" (UID: \"3a72b5be-ac50-4f83-89ab-f0664cf51f1f\") " pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.382545 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.382421    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:53:59.382545 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.382449    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmwgm\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-kube-api-access-qmwgm\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.382545 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.382481    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a72b5be-ac50-4f83-89ab-f0664cf51f1f-serving-cert\") pod \"insights-operator-5844b856db-dpdf4\" (UID: \"3a72b5be-ac50-4f83-89ab-f0664cf51f1f\") " pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.382873 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.382553    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f7c60693-7739-4de7-9a76-13ce9a464eb8-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-2v8pp\" (UID: \"f7c60693-7739-4de7-9a76-13ce9a464eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"
Sep 17 10:53:59.382873 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.382581    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-trusted-ca\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.382873 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.382640    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a72b5be-ac50-4f83-89ab-f0664cf51f1f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-dpdf4\" (UID: \"3a72b5be-ac50-4f83-89ab-f0664cf51f1f\") " pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.383054 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.383034    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle podName:3738db4a-c2eb-42b9-ad15-280d36c6b75e nodeName:}" failed. No retries permitted until 2026-09-17 10:53:59.883014293 +0000 UTC m=+33.981059844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle") pod "router-default-7fc56fb846-t46f7" (UID: "3738db4a-c2eb-42b9-ad15-280d36c6b75e") : configmap references non-existent config key: service-ca.crt
Sep 17 10:53:59.383453 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.383410    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f7c60693-7739-4de7-9a76-13ce9a464eb8-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-2v8pp\" (UID: \"f7c60693-7739-4de7-9a76-13ce9a464eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"
Sep 17 10:53:59.385238 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.384530    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-stats-auth\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:53:59.385854 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.385832    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-default-certificate\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:53:59.394975 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.394941    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a72b5be-ac50-4f83-89ab-f0664cf51f1f-tmp\") pod \"insights-operator-5844b856db-dpdf4\" (UID: \"3a72b5be-ac50-4f83-89ab-f0664cf51f1f\") " pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.395458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.395427    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a72b5be-ac50-4f83-89ab-f0664cf51f1f-service-ca-bundle\") pod \"insights-operator-5844b856db-dpdf4\" (UID: \"3a72b5be-ac50-4f83-89ab-f0664cf51f1f\") " pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.396189 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.396021    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a72b5be-ac50-4f83-89ab-f0664cf51f1f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-dpdf4\" (UID: \"3a72b5be-ac50-4f83-89ab-f0664cf51f1f\") " pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.396189 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.396078    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3a72b5be-ac50-4f83-89ab-f0664cf51f1f-snapshots\") pod \"insights-operator-5844b856db-dpdf4\" (UID: \"3a72b5be-ac50-4f83-89ab-f0664cf51f1f\") " pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.399538 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.399502    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8fd5d4c56-t84nw"]
Sep 17 10:53:59.399642 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.399549    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r"]
Sep 17 10:53:59.399642 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.399571    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8tngn"]
Sep 17 10:53:59.399642 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.399598    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-tt7px"]
Sep 17 10:53:59.399788 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.399670    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l72b\" (UniqueName: \"kubernetes.io/projected/3a72b5be-ac50-4f83-89ab-f0664cf51f1f-kube-api-access-7l72b\") pod \"insights-operator-5844b856db-dpdf4\" (UID: \"3a72b5be-ac50-4f83-89ab-f0664cf51f1f\") " pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.399788 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.399728    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-dz7s6"]
Sep 17 10:53:59.399788 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.399761    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t"]
Sep 17 10:53:59.399917 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.399792    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-frdv7"]
Sep 17 10:53:59.399917 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.399831    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq"]
Sep 17 10:53:59.399917 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.399789    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8tngn"
Sep 17 10:53:59.399917 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.399869    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7fc56fb846-t46f7"]
Sep 17 10:53:59.400084 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.399940    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jcmwx"]
Sep 17 10:53:59.401120 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.401101    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89xtc\" (UniqueName: \"kubernetes.io/projected/3738db4a-c2eb-42b9-ad15-280d36c6b75e-kube-api-access-89xtc\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:53:59.401212 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.401196    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a72b5be-ac50-4f83-89ab-f0664cf51f1f-serving-cert\") pod \"insights-operator-5844b856db-dpdf4\" (UID: \"3a72b5be-ac50-4f83-89ab-f0664cf51f1f\") " pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.401589 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.401568    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6v4g\" (UniqueName: \"kubernetes.io/projected/f7c60693-7739-4de7-9a76-13ce9a464eb8-kube-api-access-l6v4g\") pod \"cluster-monitoring-operator-6cd954b897-2v8pp\" (UID: \"f7c60693-7739-4de7-9a76-13ce9a464eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"
Sep 17 10:53:59.403650 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.403616    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 10:53:59.403759 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.403656    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bbfck\""
Sep 17 10:53:59.403759 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.403627    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 10:53:59.403908 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.403892    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 10:53:59.483657 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.483573    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.483657 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.483606    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-certificates\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.484174 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.483671    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7e8c9eb-73e6-41a6-992a-a99f6d699061-cert\") pod \"ingress-canary-8tngn\" (UID: \"d7e8c9eb-73e6-41a6-992a-a99f6d699061\") " pod="openshift-ingress-canary/ingress-canary-8tngn"
Sep 17 10:53:59.484174 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.483689    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrgn6\" (UniqueName: \"kubernetes.io/projected/b1ded66f-7439-4acc-a8e6-9a3b206d9af7-kube-api-access-hrgn6\") pod \"network-check-source-6f8b768455-tt7px\" (UID: \"b1ded66f-7439-4acc-a8e6-9a3b206d9af7\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-tt7px"
Sep 17 10:53:59.484174 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.483713    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q897g\" (UniqueName: \"kubernetes.io/projected/d7e8c9eb-73e6-41a6-992a-a99f6d699061-kube-api-access-q897g\") pod \"ingress-canary-8tngn\" (UID: \"d7e8c9eb-73e6-41a6-992a-a99f6d699061\") " pod="openshift-ingress-canary/ingress-canary-8tngn"
Sep 17 10:53:59.484174 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.483735    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e6482862-22db-436b-97ae-867b9172e290-serving-cert\") pod \"console-operator-696884fb9c-dz7s6\" (UID: \"e6482862-22db-436b-97ae-867b9172e290\") " pod="openshift-console-operator/console-operator-696884fb9c-dz7s6"
Sep 17 10:53:59.484174 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.483770    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 10:53:59.484174 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.483786    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bee64b2f-19f8-475f-8f4e-6f0017d52542-config-volume\") pod \"dns-default-jcmwx\" (UID: \"bee64b2f-19f8-475f-8f4e-6f0017d52542\") " pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:53:59.484174 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.483794    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fd5d4c56-t84nw: secret "image-registry-tls" not found
Sep 17 10:53:59.484174 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.483931    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls podName:97de9f7d-09e9-4412-9c7e-1eb4f2a4d447 nodeName:}" failed. No retries permitted until 2026-09-17 10:53:59.983908849 +0000 UTC m=+34.081954407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls") pod "image-registry-8fd5d4c56-t84nw" (UID: "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447") : secret "image-registry-tls" not found
Sep 17 10:53:59.484174 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bee64b2f-19f8-475f-8f4e-6f0017d52542-tmp-dir\") pod \"dns-default-jcmwx\" (UID: \"bee64b2f-19f8-475f-8f4e-6f0017d52542\") " pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:53:59.484174 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484127    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqkfc\" (UniqueName: \"kubernetes.io/projected/ad3b878d-f419-4e39-88d1-a53ee931a22e-kube-api-access-kqkfc\") pod \"service-ca-operator-cf7c776b7-mrd5r\" (UID: \"ad3b878d-f419-4e39-88d1-a53ee931a22e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r"
Sep 17 10:53:59.484640 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484185    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwklz\" (UniqueName: \"kubernetes.io/projected/e6482862-22db-436b-97ae-867b9172e290-kube-api-access-qwklz\") pod \"console-operator-696884fb9c-dz7s6\" (UID: \"e6482862-22db-436b-97ae-867b9172e290\") " pod="openshift-console-operator/console-operator-696884fb9c-dz7s6"
Sep 17 10:53:59.484640 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484265    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gd7x\" (UniqueName: \"kubernetes.io/projected/bee64b2f-19f8-475f-8f4e-6f0017d52542-kube-api-access-7gd7x\") pod \"dns-default-jcmwx\" (UID: \"bee64b2f-19f8-475f-8f4e-6f0017d52542\") " pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:53:59.484640 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484288    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-certificates\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.484640 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484299    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad3b878d-f419-4e39-88d1-a53ee931a22e-config\") pod \"service-ca-operator-cf7c776b7-mrd5r\" (UID: \"ad3b878d-f419-4e39-88d1-a53ee931a22e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r"
Sep 17 10:53:59.484640 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484352    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-image-registry-private-configuration\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.484640 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484399    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6482862-22db-436b-97ae-867b9172e290-trusted-ca\") pod \"console-operator-696884fb9c-dz7s6\" (UID: \"e6482862-22db-436b-97ae-867b9172e290\") " pod="openshift-console-operator/console-operator-696884fb9c-dz7s6"
Sep 17 10:53:59.484640 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484434    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d1fa8ec-467e-4787-b213-86b104ae0e61-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-z6w7t\" (UID: \"8d1fa8ec-467e-4787-b213-86b104ae0e61\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t"
Sep 17 10:53:59.484640 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484468    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hgkk\" (UniqueName: \"kubernetes.io/projected/8d1fa8ec-467e-4787-b213-86b104ae0e61-kube-api-access-6hgkk\") pod \"kube-storage-version-migrator-operator-6c8f84f897-z6w7t\" (UID: \"8d1fa8ec-467e-4787-b213-86b104ae0e61\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t"
Sep 17 10:53:59.484640 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484501    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-installation-pull-secrets\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.484640 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484523    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad3b878d-f419-4e39-88d1-a53ee931a22e-serving-cert\") pod \"service-ca-operator-cf7c776b7-mrd5r\" (UID: \"ad3b878d-f419-4e39-88d1-a53ee931a22e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r"
Sep 17 10:53:59.484640 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484548    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nq9rq\" (UID: \"d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq"
Sep 17 10:53:59.484640 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484570    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xct2n\" (UniqueName: \"kubernetes.io/projected/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-kube-api-access-xct2n\") pod \"cluster-samples-operator-86499c89c4-nq9rq\" (UID: \"d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq"
Sep 17 10:53:59.484640 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484592    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-ca-trust-extracted\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.484640 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484631    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n44vg\" (UniqueName: \"kubernetes.io/projected/c5717a5d-758e-4255-a058-84bfefce6fbc-kube-api-access-n44vg\") pod \"volume-data-source-validator-86695b7cc7-frdv7\" (UID: \"c5717a5d-758e-4255-a058-84bfefce6fbc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-frdv7"
Sep 17 10:53:59.485326 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484663    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e6482862-22db-436b-97ae-867b9172e290-config\") pod \"console-operator-696884fb9c-dz7s6\" (UID: \"e6482862-22db-436b-97ae-867b9172e290\") " pod="openshift-console-operator/console-operator-696884fb9c-dz7s6"
Sep 17 10:53:59.485326 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484723    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmwgm\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-kube-api-access-qmwgm\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.485326 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484761    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-trusted-ca\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.485326 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484798    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-bound-sa-token\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.485326 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484840    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d1fa8ec-467e-4787-b213-86b104ae0e61-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-z6w7t\" (UID: \"8d1fa8ec-467e-4787-b213-86b104ae0e61\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t"
Sep 17 10:53:59.485326 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484864    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls\") pod \"dns-default-jcmwx\" (UID: \"bee64b2f-19f8-475f-8f4e-6f0017d52542\") " pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:53:59.485326 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.484910    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad3b878d-f419-4e39-88d1-a53ee931a22e-config\") pod \"service-ca-operator-cf7c776b7-mrd5r\" (UID: \"ad3b878d-f419-4e39-88d1-a53ee931a22e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r"
Sep 17 10:53:59.485326 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.485108    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-ca-trust-extracted\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.485326 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.485158    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 10:53:59.485326 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.485277    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls podName:d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658 nodeName:}" failed. No retries permitted until 2026-09-17 10:53:59.985259035 +0000 UTC m=+34.083304594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-nq9rq" (UID: "d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658") : secret "samples-operator-tls" not found
Sep 17 10:53:59.485922 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.485904    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-trusted-ca\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.487426 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.487364    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-image-registry-private-configuration\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.487588 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.487563    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad3b878d-f419-4e39-88d1-a53ee931a22e-serving-cert\") pod \"service-ca-operator-cf7c776b7-mrd5r\" (UID: \"ad3b878d-f419-4e39-88d1-a53ee931a22e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r"
Sep 17 10:53:59.487684 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.487672    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-installation-pull-secrets\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.490596 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.490578    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-dpdf4"
Sep 17 10:53:59.503608 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.503576    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqkfc\" (UniqueName: \"kubernetes.io/projected/ad3b878d-f419-4e39-88d1-a53ee931a22e-kube-api-access-kqkfc\") pod \"service-ca-operator-cf7c776b7-mrd5r\" (UID: \"ad3b878d-f419-4e39-88d1-a53ee931a22e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r"
Sep 17 10:53:59.503757 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.503648    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrgn6\" (UniqueName: \"kubernetes.io/projected/b1ded66f-7439-4acc-a8e6-9a3b206d9af7-kube-api-access-hrgn6\") pod \"network-check-source-6f8b768455-tt7px\" (UID: \"b1ded66f-7439-4acc-a8e6-9a3b206d9af7\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-tt7px"
Sep 17 10:53:59.506052 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.506000    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n44vg\" (UniqueName: \"kubernetes.io/projected/c5717a5d-758e-4255-a058-84bfefce6fbc-kube-api-access-n44vg\") pod \"volume-data-source-validator-86695b7cc7-frdv7\" (UID: \"c5717a5d-758e-4255-a058-84bfefce6fbc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-frdv7"
Sep 17 10:53:59.506529 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.506513    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-bound-sa-token\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.507109 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.507083    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmwgm\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-kube-api-access-qmwgm\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.507671 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.507651    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xct2n\" (UniqueName: \"kubernetes.io/projected/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-kube-api-access-xct2n\") pod \"cluster-samples-operator-86499c89c4-nq9rq\" (UID: \"d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq"
Sep 17 10:53:59.563684 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.563488    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r"
Sep 17 10:53:59.581766 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.581659    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-tt7px"
Sep 17 10:53:59.586004 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.585978    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d1fa8ec-467e-4787-b213-86b104ae0e61-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-z6w7t\" (UID: \"8d1fa8ec-467e-4787-b213-86b104ae0e61\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t"
Sep 17 10:53:59.586122 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.586010    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls\") pod \"dns-default-jcmwx\" (UID: \"bee64b2f-19f8-475f-8f4e-6f0017d52542\") " pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:53:59.586122 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.586050    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7e8c9eb-73e6-41a6-992a-a99f6d699061-cert\") pod \"ingress-canary-8tngn\" (UID: \"d7e8c9eb-73e6-41a6-992a-a99f6d699061\") " pod="openshift-ingress-canary/ingress-canary-8tngn"
Sep 17 10:53:59.586122 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.586068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q897g\" (UniqueName: \"kubernetes.io/projected/d7e8c9eb-73e6-41a6-992a-a99f6d699061-kube-api-access-q897g\") pod \"ingress-canary-8tngn\" (UID: \"d7e8c9eb-73e6-41a6-992a-a99f6d699061\") " pod="openshift-ingress-canary/ingress-canary-8tngn"
Sep 17 10:53:59.586122 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.586082    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e6482862-22db-436b-97ae-867b9172e290-serving-cert\") pod \"console-operator-696884fb9c-dz7s6\" (UID: \"e6482862-22db-436b-97ae-867b9172e290\") " pod="openshift-console-operator/console-operator-696884fb9c-dz7s6"
Sep 17 10:53:59.586122 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.586104    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bee64b2f-19f8-475f-8f4e-6f0017d52542-config-volume\") pod \"dns-default-jcmwx\" (UID: \"bee64b2f-19f8-475f-8f4e-6f0017d52542\") " pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:53:59.586361 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.586161    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bee64b2f-19f8-475f-8f4e-6f0017d52542-tmp-dir\") pod \"dns-default-jcmwx\" (UID: \"bee64b2f-19f8-475f-8f4e-6f0017d52542\") " pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:53:59.586361 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.586200    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwklz\" (UniqueName: \"kubernetes.io/projected/e6482862-22db-436b-97ae-867b9172e290-kube-api-access-qwklz\") pod \"console-operator-696884fb9c-dz7s6\" (UID: \"e6482862-22db-436b-97ae-867b9172e290\") " pod="openshift-console-operator/console-operator-696884fb9c-dz7s6"
Sep 17 10:53:59.586361 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.586235    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gd7x\" (UniqueName: \"kubernetes.io/projected/bee64b2f-19f8-475f-8f4e-6f0017d52542-kube-api-access-7gd7x\") pod \"dns-default-jcmwx\" (UID: \"bee64b2f-19f8-475f-8f4e-6f0017d52542\") " pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:53:59.586361 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.586268    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6482862-22db-436b-97ae-867b9172e290-trusted-ca\") pod \"console-operator-696884fb9c-dz7s6\" (UID: \"e6482862-22db-436b-97ae-867b9172e290\") " pod="openshift-console-operator/console-operator-696884fb9c-dz7s6"
Sep 17 10:53:59.586361 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.586288    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d1fa8ec-467e-4787-b213-86b104ae0e61-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-z6w7t\" (UID: \"8d1fa8ec-467e-4787-b213-86b104ae0e61\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t"
Sep 17 10:53:59.586361 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.586305    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hgkk\" (UniqueName: \"kubernetes.io/projected/8d1fa8ec-467e-4787-b213-86b104ae0e61-kube-api-access-6hgkk\") pod \"kube-storage-version-migrator-operator-6c8f84f897-z6w7t\" (UID: \"8d1fa8ec-467e-4787-b213-86b104ae0e61\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t"
Sep 17 10:53:59.586361 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.586352    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e6482862-22db-436b-97ae-867b9172e290-config\") pod \"console-operator-696884fb9c-dz7s6\" (UID: \"e6482862-22db-436b-97ae-867b9172e290\") " pod="openshift-console-operator/console-operator-696884fb9c-dz7s6"
Sep 17 10:53:59.587387 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.586896    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 10:53:59.587387 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.586951    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls podName:bee64b2f-19f8-475f-8f4e-6f0017d52542 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:00.086937845 +0000 UTC m=+34.184983394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls") pod "dns-default-jcmwx" (UID: "bee64b2f-19f8-475f-8f4e-6f0017d52542") : secret "dns-default-metrics-tls" not found
Sep 17 10:53:59.587387 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.587141    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e6482862-22db-436b-97ae-867b9172e290-config\") pod \"console-operator-696884fb9c-dz7s6\" (UID: \"e6482862-22db-436b-97ae-867b9172e290\") " pod="openshift-console-operator/console-operator-696884fb9c-dz7s6"
Sep 17 10:53:59.587387 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.587202    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bee64b2f-19f8-475f-8f4e-6f0017d52542-tmp-dir\") pod \"dns-default-jcmwx\" (UID: \"bee64b2f-19f8-475f-8f4e-6f0017d52542\") " pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:53:59.587387 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.587228    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bee64b2f-19f8-475f-8f4e-6f0017d52542-config-volume\") pod \"dns-default-jcmwx\" (UID: \"bee64b2f-19f8-475f-8f4e-6f0017d52542\") " pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:53:59.587788 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.587558    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6482862-22db-436b-97ae-867b9172e290-trusted-ca\") pod \"console-operator-696884fb9c-dz7s6\" (UID: \"e6482862-22db-436b-97ae-867b9172e290\") " pod="openshift-console-operator/console-operator-696884fb9c-dz7s6"
Sep 17 10:53:59.587788 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.587646    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d1fa8ec-467e-4787-b213-86b104ae0e61-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-z6w7t\" (UID: \"8d1fa8ec-467e-4787-b213-86b104ae0e61\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t"
Sep 17 10:53:59.590097 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.590052    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d1fa8ec-467e-4787-b213-86b104ae0e61-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-z6w7t\" (UID: \"8d1fa8ec-467e-4787-b213-86b104ae0e61\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t"
Sep 17 10:53:59.590224 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.590207    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e6482862-22db-436b-97ae-867b9172e290-serving-cert\") pod \"console-operator-696884fb9c-dz7s6\" (UID: \"e6482862-22db-436b-97ae-867b9172e290\") " pod="openshift-console-operator/console-operator-696884fb9c-dz7s6"
Sep 17 10:53:59.590268 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.590252    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7e8c9eb-73e6-41a6-992a-a99f6d699061-cert\") pod \"ingress-canary-8tngn\" (UID: \"d7e8c9eb-73e6-41a6-992a-a99f6d699061\") " pod="openshift-ingress-canary/ingress-canary-8tngn"
Sep 17 10:53:59.599335 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.599157    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gd7x\" (UniqueName: \"kubernetes.io/projected/bee64b2f-19f8-475f-8f4e-6f0017d52542-kube-api-access-7gd7x\") pod \"dns-default-jcmwx\" (UID: \"bee64b2f-19f8-475f-8f4e-6f0017d52542\") " pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:53:59.604041 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.603965    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwklz\" (UniqueName: \"kubernetes.io/projected/e6482862-22db-436b-97ae-867b9172e290-kube-api-access-qwklz\") pod \"console-operator-696884fb9c-dz7s6\" (UID: \"e6482862-22db-436b-97ae-867b9172e290\") " pod="openshift-console-operator/console-operator-696884fb9c-dz7s6"
Sep 17 10:53:59.604041 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.603990    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q897g\" (UniqueName: \"kubernetes.io/projected/d7e8c9eb-73e6-41a6-992a-a99f6d699061-kube-api-access-q897g\") pod \"ingress-canary-8tngn\" (UID: \"d7e8c9eb-73e6-41a6-992a-a99f6d699061\") " pod="openshift-ingress-canary/ingress-canary-8tngn"
Sep 17 10:53:59.604453 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.604429    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hgkk\" (UniqueName: \"kubernetes.io/projected/8d1fa8ec-467e-4787-b213-86b104ae0e61-kube-api-access-6hgkk\") pod \"kube-storage-version-migrator-operator-6c8f84f897-z6w7t\" (UID: \"8d1fa8ec-467e-4787-b213-86b104ae0e61\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t"
Sep 17 10:53:59.633437 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.633405    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-frdv7"
Sep 17 10:53:59.648609 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.648577    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t"
Sep 17 10:53:59.667896 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.667867    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-dz7s6"
Sep 17 10:53:59.719574 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.719499    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8tngn"
Sep 17 10:53:59.825672 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.825610    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-dpdf4"]
Sep 17 10:53:59.829451 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:59.829420    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a72b5be_ac50_4f83_89ab_f0664cf51f1f.slice/crio-c2a814284f78416986e03f6d9da1b675163fde76af03e36e3cbd9526ca7562dd WatchSource:0}: Error finding container c2a814284f78416986e03f6d9da1b675163fde76af03e36e3cbd9526ca7562dd: Status 404 returned error can't find the container with id c2a814284f78416986e03f6d9da1b675163fde76af03e36e3cbd9526ca7562dd
Sep 17 10:53:59.883985 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.883801    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r"]
Sep 17 10:53:59.886956 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:59.886928    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3b878d_f419_4e39_88d1_a53ee931a22e.slice/crio-4e26b44a186f3dbf4332b5cf8c9140d6ad2c2cb7a6499934bf69407bd1035807 WatchSource:0}: Error finding container 4e26b44a186f3dbf4332b5cf8c9140d6ad2c2cb7a6499934bf69407bd1035807: Status 404 returned error can't find the container with id 4e26b44a186f3dbf4332b5cf8c9140d6ad2c2cb7a6499934bf69407bd1035807
Sep 17 10:53:59.890923 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.890860    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:53:59.890923 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.890908    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:53:59.891045 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.891015    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2v8pp\" (UID: \"f7c60693-7739-4de7-9a76-13ce9a464eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"
Sep 17 10:53:59.891045 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.891030    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 10:53:59.891142 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.891087    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs podName:3738db4a-c2eb-42b9-ad15-280d36c6b75e nodeName:}" failed. No retries permitted until 2026-09-17 10:54:00.891069554 +0000 UTC m=+34.989115102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs") pod "router-default-7fc56fb846-t46f7" (UID: "3738db4a-c2eb-42b9-ad15-280d36c6b75e") : secret "router-metrics-certs-default" not found
Sep 17 10:53:59.891195 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.891140    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 10:53:59.891195 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.891170    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle podName:3738db4a-c2eb-42b9-ad15-280d36c6b75e nodeName:}" failed. No retries permitted until 2026-09-17 10:54:00.89115864 +0000 UTC m=+34.989204201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle") pod "router-default-7fc56fb846-t46f7" (UID: "3738db4a-c2eb-42b9-ad15-280d36c6b75e") : configmap references non-existent config key: service-ca.crt
Sep 17 10:53:59.891195 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.891188    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls podName:f7c60693-7739-4de7-9a76-13ce9a464eb8 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:00.891179042 +0000 UTC m=+34.989224591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-2v8pp" (UID: "f7c60693-7739-4de7-9a76-13ce9a464eb8") : secret "cluster-monitoring-operator-tls" not found
Sep 17 10:53:59.911066 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.911036    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-tt7px"]
Sep 17 10:53:59.913952 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:59.913921    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ded66f_7439_4acc_a8e6_9a3b206d9af7.slice/crio-27fb379581b1bdbdcb42a6d8d5361bdb9ff64df4ac8ef53a4fd9ec3f90e3553c WatchSource:0}: Error finding container 27fb379581b1bdbdcb42a6d8d5361bdb9ff64df4ac8ef53a4fd9ec3f90e3553c: Status 404 returned error can't find the container with id 27fb379581b1bdbdcb42a6d8d5361bdb9ff64df4ac8ef53a4fd9ec3f90e3553c
Sep 17 10:53:59.953693 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.953661    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-frdv7"]
Sep 17 10:53:59.957114 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:59.957087    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5717a5d_758e_4255_a058_84bfefce6fbc.slice/crio-6c83172f0a732e856e581b3bb1dbdfdcd5919ebec920b695831c9937875c756b WatchSource:0}: Error finding container 6c83172f0a732e856e581b3bb1dbdfdcd5919ebec920b695831c9937875c756b: Status 404 returned error can't find the container with id 6c83172f0a732e856e581b3bb1dbdfdcd5919ebec920b695831c9937875c756b
Sep 17 10:53:59.968025 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.967995    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t"]
Sep 17 10:53:59.971410 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:59.971385    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1fa8ec_467e_4787_b213_86b104ae0e61.slice/crio-7897d506672107d778cb74de4bd7cd8826a4d3b19241a3cf155ea749a1f9591a WatchSource:0}: Error finding container 7897d506672107d778cb74de4bd7cd8826a4d3b19241a3cf155ea749a1f9591a: Status 404 returned error can't find the container with id 7897d506672107d778cb74de4bd7cd8826a4d3b19241a3cf155ea749a1f9591a
Sep 17 10:53:59.990257 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.990126    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-dz7s6"]
Sep 17 10:53:59.992139 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.992113    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nq9rq\" (UID: \"d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq"
Sep 17 10:53:59.992321 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:53:59.992298    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:53:59.992391 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.992345    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 10:53:59.992442 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.992417    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 10:53:59.992442 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.992432    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fd5d4c56-t84nw: secret "image-registry-tls" not found
Sep 17 10:53:59.992442 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.992438    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls podName:d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:00.99240098 +0000 UTC m=+35.090446529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-nq9rq" (UID: "d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658") : secret "samples-operator-tls" not found
Sep 17 10:53:59.992578 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:53:59.992479    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls podName:97de9f7d-09e9-4412-9c7e-1eb4f2a4d447 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:00.992463191 +0000 UTC m=+35.090508747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls") pod "image-registry-8fd5d4c56-t84nw" (UID: "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447") : secret "image-registry-tls" not found
Sep 17 10:53:59.993038 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:53:59.993016    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6482862_22db_436b_97ae_867b9172e290.slice/crio-cf8965251cbd718c839940fcf6c57e7a7c43f9e767b88be0a1cbf9dabe9dd632 WatchSource:0}: Error finding container cf8965251cbd718c839940fcf6c57e7a7c43f9e767b88be0a1cbf9dabe9dd632: Status 404 returned error can't find the container with id cf8965251cbd718c839940fcf6c57e7a7c43f9e767b88be0a1cbf9dabe9dd632
Sep 17 10:54:00.036832 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.036582    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8tngn"]
Sep 17 10:54:00.039420 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:54:00.039391    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e8c9eb_73e6_41a6_992a_a99f6d699061.slice/crio-41dba66c52cf72ec945e4f9afb26c102e6efc29931ac194b1159f1308fc92609 WatchSource:0}: Error finding container 41dba66c52cf72ec945e4f9afb26c102e6efc29931ac194b1159f1308fc92609: Status 404 returned error can't find the container with id 41dba66c52cf72ec945e4f9afb26c102e6efc29931ac194b1159f1308fc92609
Sep 17 10:54:00.093637 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.093603    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls\") pod \"dns-default-jcmwx\" (UID: \"bee64b2f-19f8-475f-8f4e-6f0017d52542\") " pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:54:00.093835 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:00.093800    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 10:54:00.093939 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:00.093891    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls podName:bee64b2f-19f8-475f-8f4e-6f0017d52542 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:01.093870627 +0000 UTC m=+35.191916172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls") pod "dns-default-jcmwx" (UID: "bee64b2f-19f8-475f-8f4e-6f0017d52542") : secret "dns-default-metrics-tls" not found
Sep 17 10:54:00.455341 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.454358    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:54:00.455341 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.454782    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:54:00.455341 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.454934    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:54:00.457727 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.457690    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 10:54:00.457892 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.457760    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 10:54:00.457892 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.457791    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4chbd\""
Sep 17 10:54:00.457892 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.457876    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gx69l\""
Sep 17 10:54:00.466896 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.466871    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:54:00.717360 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.717203    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-frdv7" event={"ID":"c5717a5d-758e-4255-a058-84bfefce6fbc","Type":"ContainerStarted","Data":"6c83172f0a732e856e581b3bb1dbdfdcd5919ebec920b695831c9937875c756b"}
Sep 17 10:54:00.718758 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.718688    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t" event={"ID":"8d1fa8ec-467e-4787-b213-86b104ae0e61","Type":"ContainerStarted","Data":"7897d506672107d778cb74de4bd7cd8826a4d3b19241a3cf155ea749a1f9591a"}
Sep 17 10:54:00.720378 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.720349    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-tt7px" event={"ID":"b1ded66f-7439-4acc-a8e6-9a3b206d9af7","Type":"ContainerStarted","Data":"27fb379581b1bdbdcb42a6d8d5361bdb9ff64df4ac8ef53a4fd9ec3f90e3553c"}
Sep 17 10:54:00.722163 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.722105    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r" event={"ID":"ad3b878d-f419-4e39-88d1-a53ee931a22e","Type":"ContainerStarted","Data":"4e26b44a186f3dbf4332b5cf8c9140d6ad2c2cb7a6499934bf69407bd1035807"}
Sep 17 10:54:00.723950 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.723924    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-dpdf4" event={"ID":"3a72b5be-ac50-4f83-89ab-f0664cf51f1f","Type":"ContainerStarted","Data":"c2a814284f78416986e03f6d9da1b675163fde76af03e36e3cbd9526ca7562dd"}
Sep 17 10:54:00.727048 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.726992    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8tngn" event={"ID":"d7e8c9eb-73e6-41a6-992a-a99f6d699061","Type":"ContainerStarted","Data":"41dba66c52cf72ec945e4f9afb26c102e6efc29931ac194b1159f1308fc92609"}
Sep 17 10:54:00.728975 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.728909    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-dz7s6" event={"ID":"e6482862-22db-436b-97ae-867b9172e290","Type":"ContainerStarted","Data":"cf8965251cbd718c839940fcf6c57e7a7c43f9e767b88be0a1cbf9dabe9dd632"}
Sep 17 10:54:00.901672 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.901635    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2v8pp\" (UID: \"f7c60693-7739-4de7-9a76-13ce9a464eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"
Sep 17 10:54:00.901872 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.901719    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:54:00.901872 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:00.901767    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:54:00.902058 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:00.902042    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle podName:3738db4a-c2eb-42b9-ad15-280d36c6b75e nodeName:}" failed. No retries permitted until 2026-09-17 10:54:02.902023547 +0000 UTC m=+37.000069096 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle") pod "router-default-7fc56fb846-t46f7" (UID: "3738db4a-c2eb-42b9-ad15-280d36c6b75e") : configmap references non-existent config key: service-ca.crt
Sep 17 10:54:00.902422 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:00.902402    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 10:54:00.902491 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:00.902462    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls podName:f7c60693-7739-4de7-9a76-13ce9a464eb8 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:02.902446639 +0000 UTC m=+37.000492187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-2v8pp" (UID: "f7c60693-7739-4de7-9a76-13ce9a464eb8") : secret "cluster-monitoring-operator-tls" not found
Sep 17 10:54:00.902554 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:00.902525    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 10:54:00.902610 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:00.902557    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs podName:3738db4a-c2eb-42b9-ad15-280d36c6b75e nodeName:}" failed. No retries permitted until 2026-09-17 10:54:02.90254783 +0000 UTC m=+37.000593388 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs") pod "router-default-7fc56fb846-t46f7" (UID: "3738db4a-c2eb-42b9-ad15-280d36c6b75e") : secret "router-metrics-certs-default" not found
Sep 17 10:54:01.002693 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:01.002611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nq9rq\" (UID: \"d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq"
Sep 17 10:54:01.002900 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:01.002730    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:54:01.002900 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:01.002782    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 10:54:01.002900 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:01.002881    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls podName:d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:03.002861722 +0000 UTC m=+37.100907278 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-nq9rq" (UID: "d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658") : secret "samples-operator-tls" not found
Sep 17 10:54:01.003126 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:01.002970    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 10:54:01.003126 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:01.002985    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fd5d4c56-t84nw: secret "image-registry-tls" not found
Sep 17 10:54:01.003126 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:01.003037    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls podName:97de9f7d-09e9-4412-9c7e-1eb4f2a4d447 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:03.003018535 +0000 UTC m=+37.101064082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls") pod "image-registry-8fd5d4c56-t84nw" (UID: "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447") : secret "image-registry-tls" not found
Sep 17 10:54:01.103438 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:01.103385    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls\") pod \"dns-default-jcmwx\" (UID: \"bee64b2f-19f8-475f-8f4e-6f0017d52542\") " pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:54:01.103621 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:01.103558    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 10:54:01.103688 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:01.103627    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls podName:bee64b2f-19f8-475f-8f4e-6f0017d52542 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:03.103611473 +0000 UTC m=+37.201657016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls") pod "dns-default-jcmwx" (UID: "bee64b2f-19f8-475f-8f4e-6f0017d52542") : secret "dns-default-metrics-tls" not found
Sep 17 10:54:02.841443 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:02.841406    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:02.922019 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:02.921971    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2v8pp\" (UID: \"f7c60693-7739-4de7-9a76-13ce9a464eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"
Sep 17 10:54:02.922203 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:02.922044    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:54:02.922203 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:02.922080    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:54:02.922338 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:02.922242    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle podName:3738db4a-c2eb-42b9-ad15-280d36c6b75e nodeName:}" failed. No retries permitted until 2026-09-17 10:54:06.922222487 +0000 UTC m=+41.020268049 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle") pod "router-default-7fc56fb846-t46f7" (UID: "3738db4a-c2eb-42b9-ad15-280d36c6b75e") : configmap references non-existent config key: service-ca.crt
Sep 17 10:54:02.922338 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:02.922332    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 10:54:02.922447 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:02.922371    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls podName:f7c60693-7739-4de7-9a76-13ce9a464eb8 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:06.922360339 +0000 UTC m=+41.020405903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-2v8pp" (UID: "f7c60693-7739-4de7-9a76-13ce9a464eb8") : secret "cluster-monitoring-operator-tls" not found
Sep 17 10:54:02.922502 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:02.922449    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 10:54:02.922502 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:02.922484    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs podName:3738db4a-c2eb-42b9-ad15-280d36c6b75e nodeName:}" failed. No retries permitted until 2026-09-17 10:54:06.922471168 +0000 UTC m=+41.020516733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs") pod "router-default-7fc56fb846-t46f7" (UID: "3738db4a-c2eb-42b9-ad15-280d36c6b75e") : secret "router-metrics-certs-default" not found
Sep 17 10:54:02.954215 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:02.954183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:03.023769 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:03.023061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:54:03.023769 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:03.023179    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nq9rq\" (UID: \"d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq"
Sep 17 10:54:03.023769 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:03.023365    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 10:54:03.023769 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:03.023381    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fd5d4c56-t84nw: secret "image-registry-tls" not found
Sep 17 10:54:03.023769 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:03.023437    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls podName:97de9f7d-09e9-4412-9c7e-1eb4f2a4d447 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:07.023419318 +0000 UTC m=+41.121464867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls") pod "image-registry-8fd5d4c56-t84nw" (UID: "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447") : secret "image-registry-tls" not found
Sep 17 10:54:03.024148 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:03.024034    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 10:54:03.024148 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:03.024095    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls podName:d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:07.024079362 +0000 UTC m=+41.122124924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-nq9rq" (UID: "d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658") : secret "samples-operator-tls" not found
Sep 17 10:54:03.124942 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:03.124278    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls\") pod \"dns-default-jcmwx\" (UID: \"bee64b2f-19f8-475f-8f4e-6f0017d52542\") " pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:54:03.124942 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:03.124481    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 10:54:03.124942 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:03.124555    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls podName:bee64b2f-19f8-475f-8f4e-6f0017d52542 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:07.124536687 +0000 UTC m=+41.222582237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls") pod "dns-default-jcmwx" (UID: "bee64b2f-19f8-475f-8f4e-6f0017d52542") : secret "dns-default-metrics-tls" not found
Sep 17 10:54:04.816217 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:04.816188    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:04.897328 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:04.897294    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:05.523476 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:05.523439    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:05.603335 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:05.603281    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:06.963381 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:06.963347    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:54:06.963798 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:06.963438    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2v8pp\" (UID: \"f7c60693-7739-4de7-9a76-13ce9a464eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"
Sep 17 10:54:06.963798 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:06.963477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:54:06.963798 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:06.963523    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle podName:3738db4a-c2eb-42b9-ad15-280d36c6b75e nodeName:}" failed. No retries permitted until 2026-09-17 10:54:14.963504038 +0000 UTC m=+49.061549587 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle") pod "router-default-7fc56fb846-t46f7" (UID: "3738db4a-c2eb-42b9-ad15-280d36c6b75e") : configmap references non-existent config key: service-ca.crt
Sep 17 10:54:06.963798 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:06.963615    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 10:54:06.963798 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:06.963663    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 10:54:06.963798 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:06.963701    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls podName:f7c60693-7739-4de7-9a76-13ce9a464eb8 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:14.963683897 +0000 UTC m=+49.061729456 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-2v8pp" (UID: "f7c60693-7739-4de7-9a76-13ce9a464eb8") : secret "cluster-monitoring-operator-tls" not found
Sep 17 10:54:06.963798 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:06.963721    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs podName:3738db4a-c2eb-42b9-ad15-280d36c6b75e nodeName:}" failed. No retries permitted until 2026-09-17 10:54:14.963709123 +0000 UTC m=+49.061754667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs") pod "router-default-7fc56fb846-t46f7" (UID: "3738db4a-c2eb-42b9-ad15-280d36c6b75e") : secret "router-metrics-certs-default" not found
Sep 17 10:54:07.064686 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:07.064648    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nq9rq\" (UID: \"d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq"
Sep 17 10:54:07.064844 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:07.064727    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:54:07.064844 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:07.064790    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 10:54:07.064844 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:07.064834    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 10:54:07.064844 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:07.064846    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fd5d4c56-t84nw: secret "image-registry-tls" not found
Sep 17 10:54:07.064980 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:07.064876    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls podName:d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:15.0648587 +0000 UTC m=+49.162904244 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-nq9rq" (UID: "d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658") : secret "samples-operator-tls" not found
Sep 17 10:54:07.064980 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:07.064892    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls podName:97de9f7d-09e9-4412-9c7e-1eb4f2a4d447 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:15.064885958 +0000 UTC m=+49.162931502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls") pod "image-registry-8fd5d4c56-t84nw" (UID: "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447") : secret "image-registry-tls" not found
Sep 17 10:54:07.165212 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:07.165173    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls\") pod \"dns-default-jcmwx\" (UID: \"bee64b2f-19f8-475f-8f4e-6f0017d52542\") " pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:54:07.165367 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:07.165293    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 10:54:07.165367 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:07.165344    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls podName:bee64b2f-19f8-475f-8f4e-6f0017d52542 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:15.165330463 +0000 UTC m=+49.263376006 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls") pod "dns-default-jcmwx" (UID: "bee64b2f-19f8-475f-8f4e-6f0017d52542") : secret "dns-default-metrics-tls" not found
Sep 17 10:54:07.535880 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:07.535841    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:07.634176 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:07.634135    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:08.087259 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:08.087227    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:08.165972 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:08.165938    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:08.695776 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:08.695748    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tqgcf"]
Sep 17 10:54:08.756685 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:08.756645    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-frdv7" event={"ID":"c5717a5d-758e-4255-a058-84bfefce6fbc","Type":"ContainerStarted","Data":"44f68e3c872a886548ec1aee7f50c747c2429c6fc25c5ae4f51cb7a29e341a6d"}
Sep 17 10:54:08.759835 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:08.759779    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-thr8q" event={"ID":"03e1a707-7a61-4d14-bf87-73bb46d021cd","Type":"ContainerStarted","Data":"d2b2e44baa3725a6d23ab95bee5d580b0941f9ce28fe18ad2080a1b4b5b09417"}
Sep 17 10:54:08.761348 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:08.761325    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r" event={"ID":"ad3b878d-f419-4e39-88d1-a53ee931a22e","Type":"ContainerStarted","Data":"f450038898832301540d1cb284ac408074c104e209e109a4ddec1b91b2d52278"}
Sep 17 10:54:08.763172 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:08.763153    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-dpdf4" event={"ID":"3a72b5be-ac50-4f83-89ab-f0664cf51f1f","Type":"ContainerStarted","Data":"7fe60b279d2a422eb6e2e7ec7e507844a43b5ff35973ca38c810d92f4a56e71e"}
Sep 17 10:54:08.788123 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:08.788070    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-frdv7" podStartSLOduration=11.344956448 podStartE2EDuration="19.788052439s" totalImagesPullingTime="8.443095991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:08.787489578 +0000 UTC m=+42.885535145" watchObservedRunningTime="2026-09-17 10:54:08.788052439 +0000 UTC m=+42.886098006"
Sep 17 10:54:08.795365 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:54:08.795320    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe2d05e_7ecf_4e56_8bbf_cd1bf3ad6dc5.slice/crio-01bfd3c7a1bfa346f05cd0cca0fbcdb75e95d42af8f1a8d0afd9b5a0edf23b10 WatchSource:0}: Error finding container 01bfd3c7a1bfa346f05cd0cca0fbcdb75e95d42af8f1a8d0afd9b5a0edf23b10: Status 404 returned error can't find the container with id 01bfd3c7a1bfa346f05cd0cca0fbcdb75e95d42af8f1a8d0afd9b5a0edf23b10
Sep 17 10:54:08.857223 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:08.856396    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-dpdf4" podStartSLOduration=11.230169008 podStartE2EDuration="19.856373062s" totalImagesPullingTime="8.626204054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:08.83027333 +0000 UTC m=+42.928318897" watchObservedRunningTime="2026-09-17 10:54:08.856373062 +0000 UTC m=+42.954418630"
Sep 17 10:54:08.887877 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:08.887805    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r" podStartSLOduration=11.318691376 podStartE2EDuration="19.887782978s" totalImagesPullingTime="8.569091602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:08.881797861 +0000 UTC m=+42.979843428" watchObservedRunningTime="2026-09-17 10:54:08.887782978 +0000 UTC m=+42.985828544"
Sep 17 10:54:08.937886 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:08.937792    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:09.033830 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.033775    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:09.748053 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.747956    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:09.770552 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.770515    2562 generic.go:356] "Generic (PLEG): container finished" podID="03e1a707-7a61-4d14-bf87-73bb46d021cd" containerID="d2b2e44baa3725a6d23ab95bee5d580b0941f9ce28fe18ad2080a1b4b5b09417" exitCode=0
Sep 17 10:54:09.770741 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.770599    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-thr8q" event={"ID":"03e1a707-7a61-4d14-bf87-73bb46d021cd","Type":"ContainerDied","Data":"d2b2e44baa3725a6d23ab95bee5d580b0941f9ce28fe18ad2080a1b4b5b09417"}
Sep 17 10:54:09.775718 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.775687    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-tt7px" event={"ID":"b1ded66f-7439-4acc-a8e6-9a3b206d9af7","Type":"ContainerStarted","Data":"56188ec562ca1c11995c7e885a4dbb94d9de495adb04c55519d2b639cc081fdb"}
Sep 17 10:54:09.778866 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.778811    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8tngn" event={"ID":"d7e8c9eb-73e6-41a6-992a-a99f6d699061","Type":"ContainerStarted","Data":"82c793b03769cdef71864731c3c7f0b745f04a41578c249d8566335d50b20a7a"}
Sep 17 10:54:09.781019 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.781001    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/0.log"
Sep 17 10:54:09.781134 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.781039    2562 generic.go:356] "Generic (PLEG): container finished" podID="e6482862-22db-436b-97ae-867b9172e290" containerID="98fdfc06876e107590a197641eeb6eeed2e0f948b60586a50d03e877004c7d72" exitCode=255
Sep 17 10:54:09.781281 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.781230    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-dz7s6" event={"ID":"e6482862-22db-436b-97ae-867b9172e290","Type":"ContainerDied","Data":"98fdfc06876e107590a197641eeb6eeed2e0f948b60586a50d03e877004c7d72"}
Sep 17 10:54:09.781363 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.781351    2562 scope.go:117] "RemoveContainer" containerID="98fdfc06876e107590a197641eeb6eeed2e0f948b60586a50d03e877004c7d72"
Sep 17 10:54:09.784927 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.784901    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t" event={"ID":"8d1fa8ec-467e-4787-b213-86b104ae0e61","Type":"ContainerStarted","Data":"abecc223c6a6331862538819d3756786c563b31a83ea9f3821374d081f67449b"}
Sep 17 10:54:09.787458 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.787427    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tqgcf" event={"ID":"9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5","Type":"ContainerStarted","Data":"2b27979f996472a1a5e0e889c307b7da3a900226a923e4a46f2c43444fa26ef0"}
Sep 17 10:54:09.787553 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.787465    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tqgcf" event={"ID":"9fe2d05e-7ecf-4e56-8bbf-cd1bf3ad6dc5","Type":"ContainerStarted","Data":"01bfd3c7a1bfa346f05cd0cca0fbcdb75e95d42af8f1a8d0afd9b5a0edf23b10"}
Sep 17 10:54:09.804988 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.804965    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:54:09.812853 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.812790    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8tngn" podStartSLOduration=2.365346965 podStartE2EDuration="10.812773099s" totalImagesPullingTime="8.447426134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:09.812522462 +0000 UTC m=+43.910568032" watchObservedRunningTime="2026-09-17 10:54:09.812773099 +0000 UTC m=+43.910818663"
Sep 17 10:54:09.832703 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.832614    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:09.837592 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.836259    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-tt7px" podStartSLOduration=11.927444439 podStartE2EDuration="20.836240231s" totalImagesPullingTime="8.908795792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:09.835443524 +0000 UTC m=+43.933489091" watchObservedRunningTime="2026-09-17 10:54:09.836240231 +0000 UTC m=+43.934285798"
Sep 17 10:54:09.884381 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.884320    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tqgcf" podStartSLOduration=42.884299007 podStartE2EDuration="42.884299007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:09.881876594 +0000 UTC m=+43.979922160" watchObservedRunningTime="2026-09-17 10:54:09.884299007 +0000 UTC m=+43.982344571"
Sep 17 10:54:09.901329 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:09.901094    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t" podStartSLOduration=12.39589266 podStartE2EDuration="20.901074158s" totalImagesPullingTime="8.505181498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:09.900161562 +0000 UTC m=+43.998207129" watchObservedRunningTime="2026-09-17 10:54:09.901074158 +0000 UTC m=+43.999119725"
Sep 17 10:54:10.149775 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.149735    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6vv6s"]
Sep 17 10:54:10.161015 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.160992    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:10.162258 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.162230    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:10.173578 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.173552    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6vv6s"]
Sep 17 10:54:10.173671 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.173612    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8sqhr\""
Sep 17 10:54:10.175253 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.175225    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 10:54:10.175367 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.175235    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 10:54:10.274679 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.274575    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:10.293419 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.293388    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88983730-d9d0-4e06-9224-f6920bd3f144-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:10.293555 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.293518    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hqqf\" (UniqueName: \"kubernetes.io/projected/88983730-d9d0-4e06-9224-f6920bd3f144-kube-api-access-8hqqf\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:10.293618 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.293560    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88983730-d9d0-4e06-9224-f6920bd3f144-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:10.293670 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.293626    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88983730-d9d0-4e06-9224-f6920bd3f144-data-volume\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:10.293737 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.293716    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88983730-d9d0-4e06-9224-f6920bd3f144-crio-socket\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:10.394449 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.394414    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88983730-d9d0-4e06-9224-f6920bd3f144-data-volume\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:10.394621 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.394463    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88983730-d9d0-4e06-9224-f6920bd3f144-crio-socket\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:10.394621 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.394484    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88983730-d9d0-4e06-9224-f6920bd3f144-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:10.394621 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.394565    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hqqf\" (UniqueName: \"kubernetes.io/projected/88983730-d9d0-4e06-9224-f6920bd3f144-kube-api-access-8hqqf\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:10.394621 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.394600    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88983730-d9d0-4e06-9224-f6920bd3f144-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:10.394847 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:10.394668    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 10:54:10.394847 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.394672    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88983730-d9d0-4e06-9224-f6920bd3f144-crio-socket\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:10.394847 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:10.394682    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 10:54:10.394847 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:10.394731    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88983730-d9d0-4e06-9224-f6920bd3f144-insights-runtime-extractor-tls podName:88983730-d9d0-4e06-9224-f6920bd3f144 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:10.894712971 +0000 UTC m=+44.992758534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88983730-d9d0-4e06-9224-f6920bd3f144-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6vv6s" (UID: "88983730-d9d0-4e06-9224-f6920bd3f144") : secret "insights-runtime-extractor-tls" not found
Sep 17 10:54:10.394847 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:10.394771    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88983730-d9d0-4e06-9224-f6920bd3f144-kube-rbac-proxy-cm podName:88983730-d9d0-4e06-9224-f6920bd3f144 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:10.894749048 +0000 UTC m=+44.992794605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88983730-d9d0-4e06-9224-f6920bd3f144-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6vv6s" (UID: "88983730-d9d0-4e06-9224-f6920bd3f144") : configmap "kube-rbac-proxy" not found
Sep 17 10:54:10.394847 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.394791    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88983730-d9d0-4e06-9224-f6920bd3f144-data-volume\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:10.412542 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.412506    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hqqf\" (UniqueName: \"kubernetes.io/projected/88983730-d9d0-4e06-9224-f6920bd3f144-kube-api-access-8hqqf\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:10.488892 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.488861    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:10.583911 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.583800    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:10.797645 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.797607    2562 generic.go:356] "Generic (PLEG): container finished" podID="03e1a707-7a61-4d14-bf87-73bb46d021cd" containerID="21e39a58c776c2a9ed3ea08ef18e468a7df02a9e3e912143f27806b5d30ddd78" exitCode=0
Sep 17 10:54:10.798074 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.797705    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-thr8q" event={"ID":"03e1a707-7a61-4d14-bf87-73bb46d021cd","Type":"ContainerDied","Data":"21e39a58c776c2a9ed3ea08ef18e468a7df02a9e3e912143f27806b5d30ddd78"}
Sep 17 10:54:10.799217 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.799196    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 10:54:10.799568 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.799553    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/0.log"
Sep 17 10:54:10.799652 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.799581    2562 generic.go:356] "Generic (PLEG): container finished" podID="e6482862-22db-436b-97ae-867b9172e290" containerID="ac605a9d4339e0e6fefff0da9dfa6f3b80162e2ec5bc07b66b0d94246d3e2e40" exitCode=255
Sep 17 10:54:10.799652 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.799628    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-dz7s6" event={"ID":"e6482862-22db-436b-97ae-867b9172e290","Type":"ContainerDied","Data":"ac605a9d4339e0e6fefff0da9dfa6f3b80162e2ec5bc07b66b0d94246d3e2e40"}
Sep 17 10:54:10.799746 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.799676    2562 scope.go:117] "RemoveContainer" containerID="98fdfc06876e107590a197641eeb6eeed2e0f948b60586a50d03e877004c7d72"
Sep 17 10:54:10.799900 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.799878    2562 scope.go:117] "RemoveContainer" containerID="ac605a9d4339e0e6fefff0da9dfa6f3b80162e2ec5bc07b66b0d94246d3e2e40"
Sep 17 10:54:10.800030 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:10.800014    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-dz7s6_openshift-console-operator(e6482862-22db-436b-97ae-867b9172e290)\"" pod="openshift-console-operator/console-operator-696884fb9c-dz7s6" podUID="e6482862-22db-436b-97ae-867b9172e290"
Sep 17 10:54:10.809743 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.809722    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:10.899467 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.898959    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88983730-d9d0-4e06-9224-f6920bd3f144-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:10.899467 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:10.899022    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 10:54:10.899467 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:10.899094    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88983730-d9d0-4e06-9224-f6920bd3f144-kube-rbac-proxy-cm podName:88983730-d9d0-4e06-9224-f6920bd3f144 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:11.899075831 +0000 UTC m=+45.997121377 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88983730-d9d0-4e06-9224-f6920bd3f144-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6vv6s" (UID: "88983730-d9d0-4e06-9224-f6920bd3f144") : configmap "kube-rbac-proxy" not found
Sep 17 10:54:10.899711 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:10.899561    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 10:54:10.899711 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:10.899599    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88983730-d9d0-4e06-9224-f6920bd3f144-insights-runtime-extractor-tls podName:88983730-d9d0-4e06-9224-f6920bd3f144 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:11.899588457 +0000 UTC m=+45.997634007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88983730-d9d0-4e06-9224-f6920bd3f144-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6vv6s" (UID: "88983730-d9d0-4e06-9224-f6920bd3f144") : secret "insights-runtime-extractor-tls" not found
Sep 17 10:54:10.899845 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.899801    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88983730-d9d0-4e06-9224-f6920bd3f144-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:10.920527 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:10.920489    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:11.017645 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:11.017614    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:11.806353 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:11.806321    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-thr8q" event={"ID":"03e1a707-7a61-4d14-bf87-73bb46d021cd","Type":"ContainerStarted","Data":"c2171d559e76f210764fa6ce7b97e095221df3abd7a63894d0c1800755189297"}
Sep 17 10:54:11.807833 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:11.807796    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 10:54:11.808125 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:11.808109    2562 scope.go:117] "RemoveContainer" containerID="ac605a9d4339e0e6fefff0da9dfa6f3b80162e2ec5bc07b66b0d94246d3e2e40"
Sep 17 10:54:11.808254 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:11.808238    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-dz7s6_openshift-console-operator(e6482862-22db-436b-97ae-867b9172e290)\"" pod="openshift-console-operator/console-operator-696884fb9c-dz7s6" podUID="e6482862-22db-436b-97ae-867b9172e290"
Sep 17 10:54:11.832470 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:11.832404    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-thr8q" podStartSLOduration=7.905117068 podStartE2EDuration="44.832387665s" totalImagesPullingTime="36.927270597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:53:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:11.830529744 +0000 UTC m=+45.928575309" watchObservedRunningTime="2026-09-17 10:54:11.832387665 +0000 UTC m=+45.930433233"
Sep 17 10:54:11.908788 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:11.908755    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88983730-d9d0-4e06-9224-f6920bd3f144-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:11.908959 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:11.908907    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 10:54:11.909005 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:11.908960    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88983730-d9d0-4e06-9224-f6920bd3f144-insights-runtime-extractor-tls podName:88983730-d9d0-4e06-9224-f6920bd3f144 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:13.908945509 +0000 UTC m=+48.006991058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88983730-d9d0-4e06-9224-f6920bd3f144-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6vv6s" (UID: "88983730-d9d0-4e06-9224-f6920bd3f144") : secret "insights-runtime-extractor-tls" not found
Sep 17 10:54:11.909005 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:11.908994    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88983730-d9d0-4e06-9224-f6920bd3f144-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:11.909211 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:11.909193    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 10:54:11.909295 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:11.909249    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88983730-d9d0-4e06-9224-f6920bd3f144-kube-rbac-proxy-cm podName:88983730-d9d0-4e06-9224-f6920bd3f144 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:13.909230755 +0000 UTC m=+48.007276322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88983730-d9d0-4e06-9224-f6920bd3f144-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6vv6s" (UID: "88983730-d9d0-4e06-9224-f6920bd3f144") : configmap "kube-rbac-proxy" not found
Sep 17 10:54:11.941395 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:11.941359    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:12.042614 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:12.042578    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:12.116597 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:12.116519    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:12.266928 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:12.266897    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q5vb8_16d6daf4-f47b-4282-8b60-e752eb65e772/dns-node-resolver/0.log"
Sep 17 10:54:12.584575 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:12.584525    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:12.654408 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:12.654376    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:12.867075 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:12.866999    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hlqjq_7e7c0e7e-ab26-4238-b048-869055b359b2/node-ca/0.log"
Sep 17 10:54:13.667285 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:13.667258    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8tngn_d7e8c9eb-73e6-41a6-992a-a99f6d699061/serve-healthcheck-canary/0.log"
Sep 17 10:54:13.927969 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:13.927886    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88983730-d9d0-4e06-9224-f6920bd3f144-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:13.928326 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:13.928035    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 10:54:13.928326 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:13.928076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88983730-d9d0-4e06-9224-f6920bd3f144-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:13.928326 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:13.928101    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88983730-d9d0-4e06-9224-f6920bd3f144-kube-rbac-proxy-cm podName:88983730-d9d0-4e06-9224-f6920bd3f144 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:17.928085338 +0000 UTC m=+52.026130881 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88983730-d9d0-4e06-9224-f6920bd3f144-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6vv6s" (UID: "88983730-d9d0-4e06-9224-f6920bd3f144") : configmap "kube-rbac-proxy" not found
Sep 17 10:54:13.928326 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:13.928174    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 10:54:13.928326 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:13.928221    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88983730-d9d0-4e06-9224-f6920bd3f144-insights-runtime-extractor-tls podName:88983730-d9d0-4e06-9224-f6920bd3f144 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:17.928209523 +0000 UTC m=+52.026255071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88983730-d9d0-4e06-9224-f6920bd3f144-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6vv6s" (UID: "88983730-d9d0-4e06-9224-f6920bd3f144") : secret "insights-runtime-extractor-tls" not found
Sep 17 10:54:13.972576 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:13.972545    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:15.037726 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:15.037686    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:54:15.038229 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:15.037830    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2v8pp\" (UID: \"f7c60693-7739-4de7-9a76-13ce9a464eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"
Sep 17 10:54:15.038229 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:15.037903    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle podName:3738db4a-c2eb-42b9-ad15-280d36c6b75e nodeName:}" failed. No retries permitted until 2026-09-17 10:54:31.037880466 +0000 UTC m=+65.135926010 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle") pod "router-default-7fc56fb846-t46f7" (UID: "3738db4a-c2eb-42b9-ad15-280d36c6b75e") : configmap references non-existent config key: service-ca.crt
Sep 17 10:54:15.038229 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:15.037902    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 10:54:15.038229 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:15.037946    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:54:15.038229 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:15.037961    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls podName:f7c60693-7739-4de7-9a76-13ce9a464eb8 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:31.037948818 +0000 UTC m=+65.135994362 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-2v8pp" (UID: "f7c60693-7739-4de7-9a76-13ce9a464eb8") : secret "cluster-monitoring-operator-tls" not found
Sep 17 10:54:15.038229 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:15.038052    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 10:54:15.038229 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:15.038100    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs podName:3738db4a-c2eb-42b9-ad15-280d36c6b75e nodeName:}" failed. No retries permitted until 2026-09-17 10:54:31.038086224 +0000 UTC m=+65.136131786 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs") pod "router-default-7fc56fb846-t46f7" (UID: "3738db4a-c2eb-42b9-ad15-280d36c6b75e") : secret "router-metrics-certs-default" not found
Sep 17 10:54:15.138659 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:15.138621    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:54:15.138888 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:15.138789    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 10:54:15.138888 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:15.138827    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fd5d4c56-t84nw: secret "image-registry-tls" not found
Sep 17 10:54:15.138888 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:15.138832    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nq9rq\" (UID: \"d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq"
Sep 17 10:54:15.138888 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:15.138889    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls podName:97de9f7d-09e9-4412-9c7e-1eb4f2a4d447 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:31.138871002 +0000 UTC m=+65.236916546 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls") pod "image-registry-8fd5d4c56-t84nw" (UID: "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447") : secret "image-registry-tls" not found
Sep 17 10:54:15.139047 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:15.138894    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 10:54:15.139047 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:15.138933    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls podName:d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:31.138922786 +0000 UTC m=+65.236968330 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-nq9rq" (UID: "d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658") : secret "samples-operator-tls" not found
Sep 17 10:54:15.239337 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:15.239300    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls\") pod \"dns-default-jcmwx\" (UID: \"bee64b2f-19f8-475f-8f4e-6f0017d52542\") " pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:54:15.239492 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:15.239436    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 10:54:15.239492 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:15.239488    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls podName:bee64b2f-19f8-475f-8f4e-6f0017d52542 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:31.239476245 +0000 UTC m=+65.337521790 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls") pod "dns-default-jcmwx" (UID: "bee64b2f-19f8-475f-8f4e-6f0017d52542") : secret "dns-default-metrics-tls" not found
Sep 17 10:54:15.299707 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:15.299622    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:16.654668 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:16.654631    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:16.752875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:16.752835    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:16.995545 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:16.995471    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:17.103124 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:17.103096    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:17.360750 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:17.360724    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:17.962101 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:17.962062    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88983730-d9d0-4e06-9224-f6920bd3f144-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:17.962499 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:17.962199    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88983730-d9d0-4e06-9224-f6920bd3f144-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:17.962499 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:17.962226    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 10:54:17.962499 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:17.962286    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88983730-d9d0-4e06-9224-f6920bd3f144-insights-runtime-extractor-tls podName:88983730-d9d0-4e06-9224-f6920bd3f144 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:25.962267935 +0000 UTC m=+60.060313497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88983730-d9d0-4e06-9224-f6920bd3f144-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6vv6s" (UID: "88983730-d9d0-4e06-9224-f6920bd3f144") : secret "insights-runtime-extractor-tls" not found
Sep 17 10:54:17.962705 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:17.962686    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88983730-d9d0-4e06-9224-f6920bd3f144-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:18.112112 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:18.112079    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:18.364799 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:18.364751    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:18.760452 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:18.760371    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:18.965437 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:18.965408    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:19.152122 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:19.152086    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:19.626225 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:19.626198    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:19.668618 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:19.668580    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-dz7s6"
Sep 17 10:54:19.668769 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:19.668652    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-dz7s6"
Sep 17 10:54:19.669015 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:19.669003    2562 scope.go:117] "RemoveContainer" containerID="ac605a9d4339e0e6fefff0da9dfa6f3b80162e2ec5bc07b66b0d94246d3e2e40"
Sep 17 10:54:19.669169 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:19.669151    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-dz7s6_openshift-console-operator(e6482862-22db-436b-97ae-867b9172e290)\"" pod="openshift-console-operator/console-operator-696884fb9c-dz7s6" podUID="e6482862-22db-436b-97ae-867b9172e290"
Sep 17 10:54:20.204506 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:20.204474    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:20.354423 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:20.354375    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:20.484638 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:20.484575    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:20.585024 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:20.584971    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:21.012390 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:21.012359    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:21.346572 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:21.346541    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:21.441489 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:21.441461    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:21.750492 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:21.750408    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:21.964610 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:21.964579    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:22.049498 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:22.049470    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:22.150126 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:22.150097    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:22.580242 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:22.580203    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:22.654463 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:22.654427    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:22.934900 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:22.934809    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:23.147615 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:23.147585    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:23.349397 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:23.349363    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:23.751718 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:23.751646    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:23.957090 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:23.957054    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:24.393923 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:24.393884    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:24.948564 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:24.948533    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:25.190079 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:25.190050    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:25.557831 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:25.557787    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:25.707962 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:25.707934    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lzn6w"
Sep 17 10:54:25.762438 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:25.762407    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:26.029869 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:26.029811    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88983730-d9d0-4e06-9224-f6920bd3f144-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:26.031991 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:26.031963    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88983730-d9d0-4e06-9224-f6920bd3f144-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vv6s\" (UID: \"88983730-d9d0-4e06-9224-f6920bd3f144\") " pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:26.071012 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:26.070968    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6vv6s"
Sep 17 10:54:26.143762 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:26.143737    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:26.366224 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:26.366200    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6vv6s"]
Sep 17 10:54:26.368069 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:54:26.368043    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88983730_d9d0_4e06_9224_f6920bd3f144.slice/crio-3712f2a8eeab2f2eb62a2815b1670466873a419fa22d68035da7269ab3e5515e WatchSource:0}: Error finding container 3712f2a8eeab2f2eb62a2815b1670466873a419fa22d68035da7269ab3e5515e: Status 404 returned error can't find the container with id 3712f2a8eeab2f2eb62a2815b1670466873a419fa22d68035da7269ab3e5515e
Sep 17 10:54:26.395772 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:26.395740    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:26.748657 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:26.748559    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:26.857223 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:26.857191    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6vv6s" event={"ID":"88983730-d9d0-4e06-9224-f6920bd3f144","Type":"ContainerStarted","Data":"c3b9c3cf351c8bdefe2abdcda094f787c86c56a0fd7fa2ff16a2f6c45efc3644"}
Sep 17 10:54:26.857386 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:26.857242    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6vv6s" event={"ID":"88983730-d9d0-4e06-9224-f6920bd3f144","Type":"ContainerStarted","Data":"3712f2a8eeab2f2eb62a2815b1670466873a419fa22d68035da7269ab3e5515e"}
Sep 17 10:54:27.147048 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:27.147018    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:27.349119 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:27.349091    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:27.863041 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:27.863009    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6vv6s" event={"ID":"88983730-d9d0-4e06-9224-f6920bd3f144","Type":"ContainerStarted","Data":"7161c0e92d0582968dfbc711152d1bcc036232be7bfaece76cd9c8e9034af530"}
Sep 17 10:54:28.867709 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:28.867676    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6vv6s" event={"ID":"88983730-d9d0-4e06-9224-f6920bd3f144","Type":"ContainerStarted","Data":"0a50a53bcf6d037e40b74bd18d05816c21b4b3458d677b4ca70326a63fb43c9a"}
Sep 17 10:54:28.892270 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:28.892219    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6vv6s" podStartSLOduration=16.715026612 podStartE2EDuration="18.892204541s" totalImagesPullingTime="2.177177929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:54:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:28.892033651 +0000 UTC m=+62.990079375" watchObservedRunningTime="2026-09-17 10:54:28.892204541 +0000 UTC m=+62.990250108"
Sep 17 10:54:31.073288 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.073238    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:54:31.073681 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.073338    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2v8pp\" (UID: \"f7c60693-7739-4de7-9a76-13ce9a464eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"
Sep 17 10:54:31.073681 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.073378    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:54:31.073914 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.073890    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3738db4a-c2eb-42b9-ad15-280d36c6b75e-service-ca-bundle\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:54:31.075691 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.075668    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3738db4a-c2eb-42b9-ad15-280d36c6b75e-metrics-certs\") pod \"router-default-7fc56fb846-t46f7\" (UID: \"3738db4a-c2eb-42b9-ad15-280d36c6b75e\") " pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:54:31.075796 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.075700    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7c60693-7739-4de7-9a76-13ce9a464eb8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2v8pp\" (UID: \"f7c60693-7739-4de7-9a76-13ce9a464eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"
Sep 17 10:54:31.174297 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.174257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:54:31.174436 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.174310    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs\") pod \"network-metrics-daemon-k6wfk\" (UID: \"96dd2d24-1a37-4504-b738-19be03267b6a\") " pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:54:31.174494 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.174455    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nq9rq\" (UID: \"d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq"
Sep 17 10:54:31.176577 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.176556    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls\") pod \"image-registry-8fd5d4c56-t84nw\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") " pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:54:31.176701 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.176685    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-nq9rq\" (UID: \"d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq"
Sep 17 10:54:31.177257 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.177243    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 10:54:31.186528 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.186511    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96dd2d24-1a37-4504-b738-19be03267b6a-metrics-certs\") pod \"network-metrics-daemon-k6wfk\" (UID: \"96dd2d24-1a37-4504-b738-19be03267b6a\") " pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:54:31.275876 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.275839    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls\") pod \"dns-default-jcmwx\" (UID: \"bee64b2f-19f8-475f-8f4e-6f0017d52542\") " pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:54:31.276038 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.275887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret\") pod \"global-pull-secret-syncer-csvxz\" (UID: \"df19ed65-1e04-4637-b6d6-8879cd6b750c\") " pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:54:31.278066 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.278038    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bee64b2f-19f8-475f-8f4e-6f0017d52542-metrics-tls\") pod \"dns-default-jcmwx\" (UID: \"bee64b2f-19f8-475f-8f4e-6f0017d52542\") " pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:54:31.279108 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.279093    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 10:54:31.288559 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.288540    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df19ed65-1e04-4637-b6d6-8879cd6b750c-original-pull-secret\") pod \"global-pull-secret-syncer-csvxz\" (UID: \"df19ed65-1e04-4637-b6d6-8879cd6b750c\") " pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:54:31.307339 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.307315    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7vsv8\""
Sep 17 10:54:31.314883 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.314859    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"
Sep 17 10:54:31.333039 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.333015    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xsc6h\""
Sep 17 10:54:31.340733 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.340709    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:54:31.346971 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.346952    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n4lnv\""
Sep 17 10:54:31.353786 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.353751    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:54:31.379330 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.379293    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4chbd\""
Sep 17 10:54:31.383409 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.383315    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csvxz"
Sep 17 10:54:31.386893 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.386811    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6wfk"
Sep 17 10:54:31.412843 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.412786    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-t2kxj\""
Sep 17 10:54:31.424149 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.419863    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq"
Sep 17 10:54:31.496497 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.496468    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lfdp8\""
Sep 17 10:54:31.504287 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.504259    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:54:31.610163 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.610091    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp"]
Sep 17 10:54:31.613025 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:54:31.613001    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c60693_7739_4de7_9a76_13ce9a464eb8.slice/crio-560fcf9fc40fb0e6f0e9a201c4ead7cdaa3e179a2b51c761df1922ddb44bcdc9 WatchSource:0}: Error finding container 560fcf9fc40fb0e6f0e9a201c4ead7cdaa3e179a2b51c761df1922ddb44bcdc9: Status 404 returned error can't find the container with id 560fcf9fc40fb0e6f0e9a201c4ead7cdaa3e179a2b51c761df1922ddb44bcdc9
Sep 17 10:54:31.668003 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.667979    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7fc56fb846-t46f7"]
Sep 17 10:54:31.669486 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:54:31.669462    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3738db4a_c2eb_42b9_ad15_280d36c6b75e.slice/crio-bebe56cbcf93bd6c5f78dcb574939fda4626a22bb3e8f78c0b919e5a6112aa03 WatchSource:0}: Error finding container bebe56cbcf93bd6c5f78dcb574939fda4626a22bb3e8f78c0b919e5a6112aa03: Status 404 returned error can't find the container with id bebe56cbcf93bd6c5f78dcb574939fda4626a22bb3e8f78c0b919e5a6112aa03
Sep 17 10:54:31.678797 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.678771    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8fd5d4c56-t84nw"]
Sep 17 10:54:31.682700 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:54:31.682677    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97de9f7d_09e9_4412_9c7e_1eb4f2a4d447.slice/crio-13407a0e57fe703f6c89a60607e247eb9e433762d3e09a0dca87ea4dd22b10d6 WatchSource:0}: Error finding container 13407a0e57fe703f6c89a60607e247eb9e433762d3e09a0dca87ea4dd22b10d6: Status 404 returned error can't find the container with id 13407a0e57fe703f6c89a60607e247eb9e433762d3e09a0dca87ea4dd22b10d6
Sep 17 10:54:31.711308 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.711279    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k6wfk"]
Sep 17 10:54:31.725337 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.725223    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-csvxz"]
Sep 17 10:54:31.728517 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:54:31.728484    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf19ed65_1e04_4637_b6d6_8879cd6b750c.slice/crio-bb223eedb8012e1aa78dfb810fcc5c1c616482dd78eceb9fa6ef0ab0b2dcfc87 WatchSource:0}: Error finding container bb223eedb8012e1aa78dfb810fcc5c1c616482dd78eceb9fa6ef0ab0b2dcfc87: Status 404 returned error can't find the container with id bb223eedb8012e1aa78dfb810fcc5c1c616482dd78eceb9fa6ef0ab0b2dcfc87
Sep 17 10:54:31.730677 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.730335    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq"]
Sep 17 10:54:31.787187 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.787165    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jcmwx"]
Sep 17 10:54:31.791937 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:54:31.791909    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee64b2f_19f8_475f_8f4e_6f0017d52542.slice/crio-5e5b823e052a0f2a86d8fe8050f36aff0775798d087f0946e807feeb2efda9a8 WatchSource:0}: Error finding container 5e5b823e052a0f2a86d8fe8050f36aff0775798d087f0946e807feeb2efda9a8: Status 404 returned error can't find the container with id 5e5b823e052a0f2a86d8fe8050f36aff0775798d087f0946e807feeb2efda9a8
Sep 17 10:54:31.879601 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.879524    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq" event={"ID":"d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658","Type":"ContainerStarted","Data":"86846a26065bf036032ee95ba302489fe5ebe83e94fa271d71b69f984f0c93b5"}
Sep 17 10:54:31.880743 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.880713    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k6wfk" event={"ID":"96dd2d24-1a37-4504-b738-19be03267b6a","Type":"ContainerStarted","Data":"531514164bd9ab81b195d30cdac2fca42562c3d85865c90a3828ddb1f82f8b5b"}
Sep 17 10:54:31.882448 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.882407    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7fc56fb846-t46f7" event={"ID":"3738db4a-c2eb-42b9-ad15-280d36c6b75e","Type":"ContainerStarted","Data":"6ea63e531b0b4fae7227a69b774db5054a974bab8ef52011765729af67c0a403"}
Sep 17 10:54:31.882614 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.882472    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7fc56fb846-t46f7" event={"ID":"3738db4a-c2eb-42b9-ad15-280d36c6b75e","Type":"ContainerStarted","Data":"bebe56cbcf93bd6c5f78dcb574939fda4626a22bb3e8f78c0b919e5a6112aa03"}
Sep 17 10:54:31.883617 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.883590    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jcmwx" event={"ID":"bee64b2f-19f8-475f-8f4e-6f0017d52542","Type":"ContainerStarted","Data":"5e5b823e052a0f2a86d8fe8050f36aff0775798d087f0946e807feeb2efda9a8"}
Sep 17 10:54:31.885180 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.885160    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw" event={"ID":"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447","Type":"ContainerStarted","Data":"23d6f19da8b14cade1a1b83add243dccfeed5106ae16e52d604c69753353015e"}
Sep 17 10:54:31.885280 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.885191    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw" event={"ID":"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447","Type":"ContainerStarted","Data":"13407a0e57fe703f6c89a60607e247eb9e433762d3e09a0dca87ea4dd22b10d6"}
Sep 17 10:54:31.885334 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.885319    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:54:31.886578 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.886560    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp" event={"ID":"f7c60693-7739-4de7-9a76-13ce9a464eb8","Type":"ContainerStarted","Data":"560fcf9fc40fb0e6f0e9a201c4ead7cdaa3e179a2b51c761df1922ddb44bcdc9"}
Sep 17 10:54:31.887791 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.887774    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-csvxz" event={"ID":"df19ed65-1e04-4637-b6d6-8879cd6b750c","Type":"ContainerStarted","Data":"bb223eedb8012e1aa78dfb810fcc5c1c616482dd78eceb9fa6ef0ab0b2dcfc87"}
Sep 17 10:54:31.908085 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.908027    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7fc56fb846-t46f7" podStartSLOduration=42.908006629 podStartE2EDuration="42.908006629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:31.906803505 +0000 UTC m=+66.004849071" watchObservedRunningTime="2026-09-17 10:54:31.908006629 +0000 UTC m=+66.006052196"
Sep 17 10:54:31.933808 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:31.933736    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw" podStartSLOduration=65.933715178 podStartE2EDuration="1m5.933715178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:31.932173841 +0000 UTC m=+66.030219406" watchObservedRunningTime="2026-09-17 10:54:31.933715178 +0000 UTC m=+66.031760745"
Sep 17 10:54:32.341977 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:32.341901    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:54:32.344842 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:32.344797    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:54:32.893606 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:32.893544    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:54:32.895231 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:32.895043    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7fc56fb846-t46f7"
Sep 17 10:54:33.900020 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:33.899985    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k6wfk" event={"ID":"96dd2d24-1a37-4504-b738-19be03267b6a","Type":"ContainerStarted","Data":"ed58f7d35e7a72617a6496a7db341cdd1e579cd6f8ed5d00817a850871bba539"}
Sep 17 10:54:33.900453 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:33.900045    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k6wfk" event={"ID":"96dd2d24-1a37-4504-b738-19be03267b6a","Type":"ContainerStarted","Data":"410b4dc30ffc39a4dbc56ef66b24b5b79c79d05c793b8c010d0f09593c38fdbd"}
Sep 17 10:54:33.921456 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:33.921404    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-k6wfk" podStartSLOduration=65.705654765 podStartE2EDuration="1m6.921385844s" totalImagesPullingTime="1.215731079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:53:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:33.919411157 +0000 UTC m=+68.017456723" watchObservedRunningTime="2026-09-17 10:54:33.921385844 +0000 UTC m=+68.019431411"
Sep 17 10:54:34.450012 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:34.449980    2562 scope.go:117] "RemoveContainer" containerID="ac605a9d4339e0e6fefff0da9dfa6f3b80162e2ec5bc07b66b0d94246d3e2e40"
Sep 17 10:54:34.978890 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:34.978855    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:35.093683 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:35.093647    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:35.199144 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:35.199101    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:35.311415 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:35.311387    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:37.305537 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.305502    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:37.390219 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.390131    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:37.492341 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.492306    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:37.509569 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.509538    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppd5x"]
Sep 17 10:54:37.532199 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.532158    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppd5x"]
Sep 17 10:54:37.532388 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.532290    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppd5x"
Sep 17 10:54:37.536339 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.536314    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 10:54:37.536490 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.536371    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zj2xc\""
Sep 17 10:54:37.571433 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.571399    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:37.632302 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.632264    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8d497787-3e87-4725-9e5e-8bde877d03f2-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-ppd5x\" (UID: \"8d497787-3e87-4725-9e5e-8bde877d03f2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppd5x"
Sep 17 10:54:37.733195 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.733111    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8d497787-3e87-4725-9e5e-8bde877d03f2-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-ppd5x\" (UID: \"8d497787-3e87-4725-9e5e-8bde877d03f2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppd5x"
Sep 17 10:54:37.733335 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:37.733263    2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 10:54:37.733373 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:54:37.733340    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d497787-3e87-4725-9e5e-8bde877d03f2-tls-certificates podName:8d497787-3e87-4725-9e5e-8bde877d03f2 nodeName:}" failed. No retries permitted until 2026-09-17 10:54:38.23332036 +0000 UTC m=+72.331365916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/8d497787-3e87-4725-9e5e-8bde877d03f2-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-ppd5x" (UID: "8d497787-3e87-4725-9e5e-8bde877d03f2") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 10:54:37.916469 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.916444    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 10:54:37.916638 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.916533    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-dz7s6" event={"ID":"e6482862-22db-436b-97ae-867b9172e290","Type":"ContainerStarted","Data":"b703b62a37e1eb58014ba8eb0108f7e14d349b40c0f3a26773dadc4654f1362a"}
Sep 17 10:54:37.916886 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.916865    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-dz7s6"
Sep 17 10:54:37.918730 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.918702    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jcmwx" event={"ID":"bee64b2f-19f8-475f-8f4e-6f0017d52542","Type":"ContainerStarted","Data":"59e77367ea66524df55301afd8d54441f7da868df0bc9f13fdfcf2a5b23f4315"}
Sep 17 10:54:37.918863 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.918758    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:54:37.918863 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.918782    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jcmwx" event={"ID":"bee64b2f-19f8-475f-8f4e-6f0017d52542","Type":"ContainerStarted","Data":"99ff5ab0fabb8798ef7c17880e94d6d4e50724c87bec4468d167f035a89f84db"}
Sep 17 10:54:37.920243 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.920208    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp" event={"ID":"f7c60693-7739-4de7-9a76-13ce9a464eb8","Type":"ContainerStarted","Data":"3b80259f115162cd1027d890cafc957bc44832f52c2a5aa71aaec7339485266b"}
Sep 17 10:54:37.922745 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.922718    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-csvxz" event={"ID":"df19ed65-1e04-4637-b6d6-8879cd6b750c","Type":"ContainerStarted","Data":"d63ad39f727325c82b76cfdfa55482068af389024c4309ba4ecc1537037c7a7a"}
Sep 17 10:54:37.924689 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.924664    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq" event={"ID":"d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658","Type":"ContainerStarted","Data":"b1e9464ac32054b0828bc0e0653983470b67a51dbfbc2b6dbf20978f651f6b6e"}
Sep 17 10:54:37.924799 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.924704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq" event={"ID":"d2d7c64e-a9ab-43d5-bed3-1fd8bee8b658","Type":"ContainerStarted","Data":"56673cbf3025e46aa92eed62bc5ff42517837084277db0b89178c6b53ffd7c3f"}
Sep 17 10:54:37.966128 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.966070    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-dz7s6" podStartSLOduration=40.479033009 podStartE2EDuration="48.966050868s" totalImagesPullingTime="8.487017859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:37.96522655 +0000 UTC m=+72.063272116" watchObservedRunningTime="2026-09-17 10:54:37.966050868 +0000 UTC m=+72.064096437"
Sep 17 10:54:37.974355 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.974328    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-dz7s6"
Sep 17 10:54:37.992510 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:37.992426    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jcmwx" podStartSLOduration=33.894651531 podStartE2EDuration="38.992410475s" totalImagesPullingTime="5.097758944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:37.989806522 +0000 UTC m=+72.087852089" watchObservedRunningTime="2026-09-17 10:54:37.992410475 +0000 UTC m=+72.090456040"
Sep 17 10:54:38.017913 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.017837    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2v8pp" podStartSLOduration=43.745089461 podStartE2EDuration="49.017801989s" totalImagesPullingTime="5.272712528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:38.016524917 +0000 UTC m=+72.114570479" watchObservedRunningTime="2026-09-17 10:54:38.017801989 +0000 UTC m=+72.115847553"
Sep 17 10:54:38.040521 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.040473    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-csvxz" podStartSLOduration=65.621835433 podStartE2EDuration="1m11.040456862s" totalImagesPullingTime="5.418621429s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 10:53:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:38.039227935 +0000 UTC m=+72.137273501" watchObservedRunningTime="2026-09-17 10:54:38.040456862 +0000 UTC m=+72.138502426"
Sep 17 10:54:38.062289 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.062238    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-nq9rq" podStartSLOduration=43.949718269 podStartE2EDuration="49.062220981s" totalImagesPullingTime="5.112502712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:38.061314655 +0000 UTC m=+72.159360222" watchObservedRunningTime="2026-09-17 10:54:38.062220981 +0000 UTC m=+72.160266543"
Sep 17 10:54:38.175563 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.175530    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-vd5xp"]
Sep 17 10:54:38.201551 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.201524    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-vd5xp"]
Sep 17 10:54:38.201708 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.201611    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-vd5xp"
Sep 17 10:54:38.208407 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.208378    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 10:54:38.208539 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.208512    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 10:54:38.208612 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.208512    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-nhf5w\""
Sep 17 10:54:38.238133 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.238101    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:38.238397 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.238380    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8d497787-3e87-4725-9e5e-8bde877d03f2-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-ppd5x\" (UID: \"8d497787-3e87-4725-9e5e-8bde877d03f2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppd5x"
Sep 17 10:54:38.240699 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.240680    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8d497787-3e87-4725-9e5e-8bde877d03f2-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-ppd5x\" (UID: \"8d497787-3e87-4725-9e5e-8bde877d03f2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppd5x"
Sep 17 10:54:38.339321 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.339276    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7kdc\" (UniqueName: \"kubernetes.io/projected/91f0be6e-7a30-4151-a73a-9c0af5212dc9-kube-api-access-n7kdc\") pod \"downloads-76f99584f4-vd5xp\" (UID: \"91f0be6e-7a30-4151-a73a-9c0af5212dc9\") " pod="openshift-console/downloads-76f99584f4-vd5xp"
Sep 17 10:54:38.350949 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.350919    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:38.440522 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.440479    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7kdc\" (UniqueName: \"kubernetes.io/projected/91f0be6e-7a30-4151-a73a-9c0af5212dc9-kube-api-access-n7kdc\") pod \"downloads-76f99584f4-vd5xp\" (UID: \"91f0be6e-7a30-4151-a73a-9c0af5212dc9\") " pod="openshift-console/downloads-76f99584f4-vd5xp"
Sep 17 10:54:38.442432 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.442409    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppd5x"
Sep 17 10:54:38.454648 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.454627    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7kdc\" (UniqueName: \"kubernetes.io/projected/91f0be6e-7a30-4151-a73a-9c0af5212dc9-kube-api-access-n7kdc\") pod \"downloads-76f99584f4-vd5xp\" (UID: \"91f0be6e-7a30-4151-a73a-9c0af5212dc9\") " pod="openshift-console/downloads-76f99584f4-vd5xp"
Sep 17 10:54:38.511533 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.511501    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-vd5xp"
Sep 17 10:54:38.554990 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.554963    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:38.732922 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.732892    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppd5x"]
Sep 17 10:54:38.735707 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:54:38.735681    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d497787_3e87_4725_9e5e_8bde877d03f2.slice/crio-a59908e52633716af71aa7e80957d88016c0f3ed89b0404d23f55b96f09c8541 WatchSource:0}: Error finding container a59908e52633716af71aa7e80957d88016c0f3ed89b0404d23f55b96f09c8541: Status 404 returned error can't find the container with id a59908e52633716af71aa7e80957d88016c0f3ed89b0404d23f55b96f09c8541
Sep 17 10:54:38.818954 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.818827    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-vd5xp"]
Sep 17 10:54:38.821455 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:54:38.821428    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f0be6e_7a30_4151_a73a_9c0af5212dc9.slice/crio-ab6d628c1f3c613b8e1a0a6bbbe26566fca9078a39a381b93ac6098ad8dc9ac3 WatchSource:0}: Error finding container ab6d628c1f3c613b8e1a0a6bbbe26566fca9078a39a381b93ac6098ad8dc9ac3: Status 404 returned error can't find the container with id ab6d628c1f3c613b8e1a0a6bbbe26566fca9078a39a381b93ac6098ad8dc9ac3
Sep 17 10:54:38.841229 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.841152    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:38.911747 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.911704    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:38.929701 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.929668    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-vd5xp" event={"ID":"91f0be6e-7a30-4151-a73a-9c0af5212dc9","Type":"ContainerStarted","Data":"ab6d628c1f3c613b8e1a0a6bbbe26566fca9078a39a381b93ac6098ad8dc9ac3"}
Sep 17 10:54:38.930873 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:38.930851    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppd5x" event={"ID":"8d497787-3e87-4725-9e5e-8bde877d03f2","Type":"ContainerStarted","Data":"a59908e52633716af71aa7e80957d88016c0f3ed89b0404d23f55b96f09c8541"}
Sep 17 10:54:39.048117 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:39.048082    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:39.128313 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:39.128229    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:39.262227 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:39.262196    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:39.330700 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:39.330667    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:39.858353 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:39.858317    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:40.075748 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:40.075699    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:40.249926 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:40.249829    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:40.333515 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:40.333465    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:40.444155 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:40.444110    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:40.545488 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:40.545448    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:40.647032 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:40.647002    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:40.720256 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:40.720223    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:40.941877 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:40.941767    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppd5x" event={"ID":"8d497787-3e87-4725-9e5e-8bde877d03f2","Type":"ContainerStarted","Data":"cd80aa62a43613cc1d314941bd7c397416bf69cfa1e81db5d5fcf0a7e94f9616"}
Sep 17 10:54:40.942324 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:40.942069    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppd5x"
Sep 17 10:54:40.947636 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:40.947614    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppd5x"
Sep 17 10:54:40.963830 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:40.963769    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ppd5x" podStartSLOduration=2.086068477 podStartE2EDuration="3.963749286s" totalImagesPullingTime="1.877680809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:54:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:40.961130853 +0000 UTC m=+75.059176419" watchObservedRunningTime="2026-09-17 10:54:40.963749286 +0000 UTC m=+75.061794854"
Sep 17 10:54:41.512095 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:41.512061    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:41.620235 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:41.620200    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:41.810871 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:41.810837    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tqgcf"
Sep 17 10:54:41.966102 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:41.966070    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:44.079569 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:44.079527    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:44.181547 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:44.181514    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:44.290877 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:44.290832    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:44.405315 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:44.405219    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:44.695196 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:44.695103    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:45.769503 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:45.769465    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:46.114954 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.114924    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t67nf"]
Sep 17 10:54:46.119067 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.119040    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.123099 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.123074    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 10:54:46.123627 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.123602    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 10:54:46.124324 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.124292    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 10:54:46.124324 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.124309    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kt2bx\""
Sep 17 10:54:46.124478 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.124403    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 10:54:46.160122 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.160092    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:46.210724 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.210691    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-node-exporter-tls\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.210931 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.210735    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-root\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.210931 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.210767    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-node-exporter-accelerators-collector-config\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.211042 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.210984    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-metrics-client-ca\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.211042 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.211027    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d794h\" (UniqueName: \"kubernetes.io/projected/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-kube-api-access-d794h\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.211139 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.211058    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-sys\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.211139 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.211118    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.211235 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.211187    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-node-exporter-textfile\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.211279 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.211232    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-node-exporter-wtmp\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.249215 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.249179    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:46.312325 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.312290    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-node-exporter-tls\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.312325 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.312334    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-root\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.312578 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.312355    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-node-exporter-accelerators-collector-config\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.312578 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.312382    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-metrics-client-ca\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.312578 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.312406    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d794h\" (UniqueName: \"kubernetes.io/projected/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-kube-api-access-d794h\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.312578 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.312425    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-root\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.312768 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.312733    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-sys\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.312848 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.312790    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.312910 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.312870    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-node-exporter-textfile\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.312959 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.312915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-node-exporter-wtmp\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.313094 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.313065    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-node-exporter-accelerators-collector-config\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.313094 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.313077    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-node-exporter-wtmp\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.313094 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.312870    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-sys\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.313094 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.313089    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-metrics-client-ca\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.313316 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.313163    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-node-exporter-textfile\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.315088 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.315067    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-node-exporter-tls\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.315214 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.315196    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.326256 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.326232    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d794h\" (UniqueName: \"kubernetes.io/projected/bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c-kube-api-access-d794h\") pod \"node-exporter-t67nf\" (UID: \"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c\") " pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.349165 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.349127    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:46.431182 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.431078    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t67nf"
Sep 17 10:54:46.443988 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:54:46.443952    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc5bbc0_4413_4993_b1e8_b5f7fdd5cf4c.slice/crio-0d28b6d4a95bacf96b93ed46e36ba3ca7edb8433672bc4bb2eabeae6ea404d73 WatchSource:0}: Error finding container 0d28b6d4a95bacf96b93ed46e36ba3ca7edb8433672bc4bb2eabeae6ea404d73: Status 404 returned error can't find the container with id 0d28b6d4a95bacf96b93ed46e36ba3ca7edb8433672bc4bb2eabeae6ea404d73
Sep 17 10:54:46.452395 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.452165    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:46.552228 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.552191    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:46.963405 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.963365    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:46.969973 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:46.969943    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t67nf" event={"ID":"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c","Type":"ContainerStarted","Data":"0d28b6d4a95bacf96b93ed46e36ba3ca7edb8433672bc4bb2eabeae6ea404d73"}
Sep 17 10:54:47.157718 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.157684    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:47.267277 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.267181    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 10:54:47.267966 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.267878    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:47.271576 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.271552    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.276303 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.275983    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 10:54:47.276303 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.276023    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 10:54:47.276303 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.276038    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 10:54:47.276303 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.276136    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 10:54:47.276303 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.276201    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 10:54:47.276303 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.276204    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 10:54:47.276303 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.275941    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 10:54:47.276701 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.276454    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2m2jm\""
Sep 17 10:54:47.276701 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.276654    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 10:54:47.277093 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.277052    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 10:54:47.301032 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.300999    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 10:54:47.423704 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.423664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/266ba384-8031-4300-b02f-9331659c09e2-config-out\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.423918 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.423726    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/266ba384-8031-4300-b02f-9331659c09e2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.423918 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.423760    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.423918 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.423792    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.424483 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.424455    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-config-volume\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.424628 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.424503    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.424628 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.424555    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-web-config\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.424628 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.424586    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/266ba384-8031-4300-b02f-9331659c09e2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.424628 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.424619    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/266ba384-8031-4300-b02f-9331659c09e2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.424902 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.424645    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.424902 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.424684    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.424902 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.424718    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/266ba384-8031-4300-b02f-9331659c09e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.424902 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.424756    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4pjq\" (UniqueName: \"kubernetes.io/projected/266ba384-8031-4300-b02f-9331659c09e2-kube-api-access-n4pjq\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.526065 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.525987    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-config-volume\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.526065 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.526033    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.526303 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.526255    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-web-config\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.526365 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.526348    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/266ba384-8031-4300-b02f-9331659c09e2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.526419 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.526389    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/266ba384-8031-4300-b02f-9331659c09e2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.526471 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.526417    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.526471 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.526460    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.526572 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.526486    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/266ba384-8031-4300-b02f-9331659c09e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.526572 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.526516    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4pjq\" (UniqueName: \"kubernetes.io/projected/266ba384-8031-4300-b02f-9331659c09e2-kube-api-access-n4pjq\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.526572 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.526567    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/266ba384-8031-4300-b02f-9331659c09e2-config-out\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.526719 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.526595    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/266ba384-8031-4300-b02f-9331659c09e2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.526719 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.526630    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.526719 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.526653    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.526907 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.526757    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/266ba384-8031-4300-b02f-9331659c09e2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.528061 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.527678    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/266ba384-8031-4300-b02f-9331659c09e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.529632 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.529605    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/266ba384-8031-4300-b02f-9331659c09e2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.531466 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.531409    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.531918 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.531883    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-web-config\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.532314 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.532281    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.534112 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.534087    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.534212 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.534126    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.534515 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.534485    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/266ba384-8031-4300-b02f-9331659c09e2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.534618 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.534605    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-config-volume\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.534680 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.534661    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.536451 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.536429    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/266ba384-8031-4300-b02f-9331659c09e2-config-out\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.539045 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.539022    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4pjq\" (UniqueName: \"kubernetes.io/projected/266ba384-8031-4300-b02f-9331659c09e2-kube-api-access-n4pjq\") pod \"alertmanager-main-0\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.583898 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.583867    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:54:47.934101 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.934073    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jcmwx"
Sep 17 10:54:47.976827 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.976788    2562 generic.go:356] "Generic (PLEG): container finished" podID="bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c" containerID="25f94ac396a067ee9cc45753e8505d325c955f35b212eaf43cc4fac7268feaeb" exitCode=0
Sep 17 10:54:47.977268 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.976867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t67nf" event={"ID":"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c","Type":"ContainerDied","Data":"25f94ac396a067ee9cc45753e8505d325c955f35b212eaf43cc4fac7268feaeb"}
Sep 17 10:54:47.998125 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:47.997973    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 10:54:48.004035 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:54:48.004006    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266ba384_8031_4300_b02f_9331659c09e2.slice/crio-435563c1f06c3c1de7ef45714c382f4294e6280aecfd12509d8cc1a73c6a5e76 WatchSource:0}: Error finding container 435563c1f06c3c1de7ef45714c382f4294e6280aecfd12509d8cc1a73c6a5e76: Status 404 returned error can't find the container with id 435563c1f06c3c1de7ef45714c382f4294e6280aecfd12509d8cc1a73c6a5e76
Sep 17 10:54:48.367650 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.367577    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b86b9c5f-j7sbm"]
Sep 17 10:54:48.370635 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.370617    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.373286 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.373262    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 10:54:48.373446 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.373426    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 10:54:48.374013 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.373994    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 10:54:48.374448 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.374425    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qxbsw\""
Sep 17 10:54:48.374530 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.374427    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 10:54:48.374530 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.374464    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 10:54:48.389933 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.389909    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b86b9c5f-j7sbm"]
Sep 17 10:54:48.536694 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.536626    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgrm9\" (UniqueName: \"kubernetes.io/projected/6e680891-abe6-43a0-8fe6-22edba5ff5d4-kube-api-access-lgrm9\") pod \"console-5b86b9c5f-j7sbm\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") " pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.536860 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.536767    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e680891-abe6-43a0-8fe6-22edba5ff5d4-console-oauth-config\") pod \"console-5b86b9c5f-j7sbm\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") " pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.536860 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.536807    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e680891-abe6-43a0-8fe6-22edba5ff5d4-service-ca\") pod \"console-5b86b9c5f-j7sbm\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") " pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.537027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.537001    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e680891-abe6-43a0-8fe6-22edba5ff5d4-console-config\") pod \"console-5b86b9c5f-j7sbm\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") " pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.537165 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.537059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e680891-abe6-43a0-8fe6-22edba5ff5d4-oauth-serving-cert\") pod \"console-5b86b9c5f-j7sbm\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") " pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.537165 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.537157    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e680891-abe6-43a0-8fe6-22edba5ff5d4-console-serving-cert\") pod \"console-5b86b9c5f-j7sbm\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") " pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.638306 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.638224    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e680891-abe6-43a0-8fe6-22edba5ff5d4-console-oauth-config\") pod \"console-5b86b9c5f-j7sbm\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") " pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.638306 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.638266    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e680891-abe6-43a0-8fe6-22edba5ff5d4-service-ca\") pod \"console-5b86b9c5f-j7sbm\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") " pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.638306 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.638306    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e680891-abe6-43a0-8fe6-22edba5ff5d4-console-config\") pod \"console-5b86b9c5f-j7sbm\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") " pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.638572 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.638348    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e680891-abe6-43a0-8fe6-22edba5ff5d4-oauth-serving-cert\") pod \"console-5b86b9c5f-j7sbm\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") " pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.638572 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.638401    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e680891-abe6-43a0-8fe6-22edba5ff5d4-console-serving-cert\") pod \"console-5b86b9c5f-j7sbm\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") " pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.638572 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.638427    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgrm9\" (UniqueName: \"kubernetes.io/projected/6e680891-abe6-43a0-8fe6-22edba5ff5d4-kube-api-access-lgrm9\") pod \"console-5b86b9c5f-j7sbm\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") " pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.639201 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.639156    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e680891-abe6-43a0-8fe6-22edba5ff5d4-service-ca\") pod \"console-5b86b9c5f-j7sbm\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") " pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.639318 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.639218    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e680891-abe6-43a0-8fe6-22edba5ff5d4-console-config\") pod \"console-5b86b9c5f-j7sbm\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") " pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.639364 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.639345    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e680891-abe6-43a0-8fe6-22edba5ff5d4-oauth-serving-cert\") pod \"console-5b86b9c5f-j7sbm\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") " pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.641086 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.641062    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e680891-abe6-43a0-8fe6-22edba5ff5d4-console-oauth-config\") pod \"console-5b86b9c5f-j7sbm\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") " pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.641534 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.641514    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e680891-abe6-43a0-8fe6-22edba5ff5d4-console-serving-cert\") pod \"console-5b86b9c5f-j7sbm\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") " pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.648779 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.648754    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgrm9\" (UniqueName: \"kubernetes.io/projected/6e680891-abe6-43a0-8fe6-22edba5ff5d4-kube-api-access-lgrm9\") pod \"console-5b86b9c5f-j7sbm\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") " pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.680762 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.680724    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:54:48.983104 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.982938    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"266ba384-8031-4300-b02f-9331659c09e2","Type":"ContainerStarted","Data":"435563c1f06c3c1de7ef45714c382f4294e6280aecfd12509d8cc1a73c6a5e76"}
Sep 17 10:54:48.985595 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.985561    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t67nf" event={"ID":"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c","Type":"ContainerStarted","Data":"8b0b60f0a99857b8c2533f8a2d210f84c11af9b7b2a565d3b3914d7b23bbd8be"}
Sep 17 10:54:48.985712 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:48.985620    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t67nf" event={"ID":"bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c","Type":"ContainerStarted","Data":"4bc83d072c4853b7a895f1f5c20a849468fa650c8bad0ab9df22017b48be8220"}
Sep 17 10:54:49.010675 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.010626    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t67nf" podStartSLOduration=1.640790282 podStartE2EDuration="3.0106078s" totalImagesPullingTime="1.296639511s" totalInitContainerRuntime="73.178007ms" isStatefulPod=true podCreationTimestamp="2026-09-17 10:54:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:49.009949845 +0000 UTC m=+83.107995425" watchObservedRunningTime="2026-09-17 10:54:49.0106078 +0000 UTC m=+83.108653370"
Sep 17 10:54:49.056715 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.056682    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7fc69f886-q7tdw"]
Sep 17 10:54:49.060443 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.060420    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.063454 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.063427    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 10:54:49.063576 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.063467    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8qfcmnthe07e7\""
Sep 17 10:54:49.063645 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.063632    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 10:54:49.063729 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.063711    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 10:54:49.063912 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.063892    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pq9rb\""
Sep 17 10:54:49.064003 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.063959    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 10:54:49.064118 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.064094    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 10:54:49.076431 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.076406    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7fc69f886-q7tdw"]
Sep 17 10:54:49.143480 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.143443    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/126afa14-cfb2-47cc-b856-23679e74bf47-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.143668 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.143597    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/126afa14-cfb2-47cc-b856-23679e74bf47-secret-grpc-tls\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.144171 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.143738    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/126afa14-cfb2-47cc-b856-23679e74bf47-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.144171 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.143789    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tdlr\" (UniqueName: \"kubernetes.io/projected/126afa14-cfb2-47cc-b856-23679e74bf47-kube-api-access-4tdlr\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.144171 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.143883    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/126afa14-cfb2-47cc-b856-23679e74bf47-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.144171 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.143919    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/126afa14-cfb2-47cc-b856-23679e74bf47-metrics-client-ca\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.144171 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.143963    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/126afa14-cfb2-47cc-b856-23679e74bf47-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.144171 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.144024    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/126afa14-cfb2-47cc-b856-23679e74bf47-secret-thanos-querier-tls\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.167035 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.167000    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:49.244943 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.244808    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/126afa14-cfb2-47cc-b856-23679e74bf47-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.244943 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.244881    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tdlr\" (UniqueName: \"kubernetes.io/projected/126afa14-cfb2-47cc-b856-23679e74bf47-kube-api-access-4tdlr\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.244943 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.244912    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/126afa14-cfb2-47cc-b856-23679e74bf47-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.244943 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.244941    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/126afa14-cfb2-47cc-b856-23679e74bf47-metrics-client-ca\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.245335 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.244971    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/126afa14-cfb2-47cc-b856-23679e74bf47-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.245335 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.245267    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/126afa14-cfb2-47cc-b856-23679e74bf47-secret-thanos-querier-tls\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.245442 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.245352    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/126afa14-cfb2-47cc-b856-23679e74bf47-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.245442 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.245405    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/126afa14-cfb2-47cc-b856-23679e74bf47-secret-grpc-tls\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.245885 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.245857    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/126afa14-cfb2-47cc-b856-23679e74bf47-metrics-client-ca\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.248301 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.248265    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/126afa14-cfb2-47cc-b856-23679e74bf47-secret-thanos-querier-tls\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.248301 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.248288    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/126afa14-cfb2-47cc-b856-23679e74bf47-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.248467 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.248312    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/126afa14-cfb2-47cc-b856-23679e74bf47-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.248782 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.248758    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/126afa14-cfb2-47cc-b856-23679e74bf47-secret-grpc-tls\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.248886 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.248839    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/126afa14-cfb2-47cc-b856-23679e74bf47-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.248945 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.248903    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/126afa14-cfb2-47cc-b856-23679e74bf47-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.261103 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.261079    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tdlr\" (UniqueName: \"kubernetes.io/projected/126afa14-cfb2-47cc-b856-23679e74bf47-kube-api-access-4tdlr\") pod \"thanos-querier-7fc69f886-q7tdw\" (UID: \"126afa14-cfb2-47cc-b856-23679e74bf47\") " pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.275290 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.275263    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:49.368029 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.367990    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:49.373071 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.373047    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:54:49.521147 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.521058    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:49.622656 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:49.622622    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:50.793250 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:50.793213    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-dht2n"]
Sep 17 10:54:50.795350 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:50.795314    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:50.796697 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:50.796676    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-dht2n"
Sep 17 10:54:50.817185 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:50.817149    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 10:54:50.817396 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:50.817155    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xksm2\""
Sep 17 10:54:50.817655 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:50.817633    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-dht2n"]
Sep 17 10:54:50.921559 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:50.921516    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:50.962276 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:50.962186    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q62bd\" (UniqueName: \"kubernetes.io/projected/bda08aa0-553a-4c83-9f6f-fd182889e48f-kube-api-access-q62bd\") pod \"monitoring-plugin-5d85778b9d-dht2n\" (UID: \"bda08aa0-553a-4c83-9f6f-fd182889e48f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-dht2n"
Sep 17 10:54:50.962276 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:50.962277    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bda08aa0-553a-4c83-9f6f-fd182889e48f-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-dht2n\" (UID: \"bda08aa0-553a-4c83-9f6f-fd182889e48f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-dht2n"
Sep 17 10:54:51.022091 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.022045    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:51.063255 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.063175    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q62bd\" (UniqueName: \"kubernetes.io/projected/bda08aa0-553a-4c83-9f6f-fd182889e48f-kube-api-access-q62bd\") pod \"monitoring-plugin-5d85778b9d-dht2n\" (UID: \"bda08aa0-553a-4c83-9f6f-fd182889e48f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-dht2n"
Sep 17 10:54:51.063255 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.063245    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bda08aa0-553a-4c83-9f6f-fd182889e48f-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-dht2n\" (UID: \"bda08aa0-553a-4c83-9f6f-fd182889e48f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-dht2n"
Sep 17 10:54:51.066126 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.066102    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bda08aa0-553a-4c83-9f6f-fd182889e48f-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-dht2n\" (UID: \"bda08aa0-553a-4c83-9f6f-fd182889e48f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-dht2n"
Sep 17 10:54:51.090432 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.090405    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q62bd\" (UniqueName: \"kubernetes.io/projected/bda08aa0-553a-4c83-9f6f-fd182889e48f-kube-api-access-q62bd\") pod \"monitoring-plugin-5d85778b9d-dht2n\" (UID: \"bda08aa0-553a-4c83-9f6f-fd182889e48f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-dht2n"
Sep 17 10:54:51.109201 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.109169    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-dht2n"
Sep 17 10:54:51.275226 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.275190    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"]
Sep 17 10:54:51.279145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.279121    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.283527 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.283500    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 10:54:51.283527 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.283517    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 10:54:51.283780 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.283765    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 10:54:51.283848 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.283780    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 10:54:51.283889 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.283785    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 10:54:51.284329 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.284309    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jflht\""
Sep 17 10:54:51.288639 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.288617    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 10:54:51.301059 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.301033    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"]
Sep 17 10:54:51.466655 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.466558    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/98d9144c-4704-42e8-b203-dcf14e23974a-telemeter-client-tls\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.466655 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.466609    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-682md\" (UniqueName: \"kubernetes.io/projected/98d9144c-4704-42e8-b203-dcf14e23974a-kube-api-access-682md\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.466911 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.466657    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/98d9144c-4704-42e8-b203-dcf14e23974a-federate-client-tls\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.466911 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.466748    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/98d9144c-4704-42e8-b203-dcf14e23974a-secret-telemeter-client\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.466911 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.466799    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98d9144c-4704-42e8-b203-dcf14e23974a-serving-certs-ca-bundle\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.466911 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.466876    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98d9144c-4704-42e8-b203-dcf14e23974a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.467114 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.466936    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98d9144c-4704-42e8-b203-dcf14e23974a-metrics-client-ca\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.467114 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.466966    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98d9144c-4704-42e8-b203-dcf14e23974a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.568112 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.568072    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98d9144c-4704-42e8-b203-dcf14e23974a-serving-certs-ca-bundle\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.568286 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.568134    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98d9144c-4704-42e8-b203-dcf14e23974a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.568286 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.568174    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98d9144c-4704-42e8-b203-dcf14e23974a-metrics-client-ca\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.568286 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.568202    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98d9144c-4704-42e8-b203-dcf14e23974a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.568286 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.568262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/98d9144c-4704-42e8-b203-dcf14e23974a-telemeter-client-tls\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.568486 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.568291    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-682md\" (UniqueName: \"kubernetes.io/projected/98d9144c-4704-42e8-b203-dcf14e23974a-kube-api-access-682md\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.568486 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.568329    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/98d9144c-4704-42e8-b203-dcf14e23974a-federate-client-tls\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.568486 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.568366    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/98d9144c-4704-42e8-b203-dcf14e23974a-secret-telemeter-client\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.569096 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.569060    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98d9144c-4704-42e8-b203-dcf14e23974a-serving-certs-ca-bundle\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.569324 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.569295    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98d9144c-4704-42e8-b203-dcf14e23974a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.569793 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.569770    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98d9144c-4704-42e8-b203-dcf14e23974a-metrics-client-ca\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.571326 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.571290    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98d9144c-4704-42e8-b203-dcf14e23974a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.571326 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.571320    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/98d9144c-4704-42e8-b203-dcf14e23974a-secret-telemeter-client\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.571496 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.571478    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/98d9144c-4704-42e8-b203-dcf14e23974a-federate-client-tls\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.571586 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.571565    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/98d9144c-4704-42e8-b203-dcf14e23974a-telemeter-client-tls\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.578538 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.578514    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-682md\" (UniqueName: \"kubernetes.io/projected/98d9144c-4704-42e8-b203-dcf14e23974a-kube-api-access-682md\") pod \"telemeter-client-58c97c79fd-zzkqt\" (UID: \"98d9144c-4704-42e8-b203-dcf14e23974a\") " pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:51.591605 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:51.591583    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"
Sep 17 10:54:52.394414 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:52.392564    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:52.485045 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:52.485009    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:52.589145 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:52.589112    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:52.897506 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:52.897477    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:54:53.226027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:53.225948    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:53.303860 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:53.303804    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:53.522529 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:53.522449    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:53.771643 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:53.771610    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:54.623073 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:54.623028    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:54.697400 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:54.697366    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:55.682976 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:55.682947    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-dht2n"]
Sep 17 10:54:55.684744 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:54:55.684712    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda08aa0_553a_4c83_9f6f_fd182889e48f.slice/crio-347edf969bdf919b737b9398c1f02207ac707731132a75e34b2101e5150b9019 WatchSource:0}: Error finding container 347edf969bdf919b737b9398c1f02207ac707731132a75e34b2101e5150b9019: Status 404 returned error can't find the container with id 347edf969bdf919b737b9398c1f02207ac707731132a75e34b2101e5150b9019
Sep 17 10:54:55.692674 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:55.692221    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7fc69f886-q7tdw"]
Sep 17 10:54:55.694520 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:55.694449    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-58c97c79fd-zzkqt"]
Sep 17 10:54:55.698387 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:55.698363    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b86b9c5f-j7sbm"]
Sep 17 10:54:55.698734 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:54:55.698633    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126afa14_cfb2_47cc_b856_23679e74bf47.slice/crio-117c63a9dc635250a4137e787a7759212ba2493d31433583730481472c6b19e7 WatchSource:0}: Error finding container 117c63a9dc635250a4137e787a7759212ba2493d31433583730481472c6b19e7: Status 404 returned error can't find the container with id 117c63a9dc635250a4137e787a7759212ba2493d31433583730481472c6b19e7
Sep 17 10:54:55.704097 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:54:55.703878    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e680891_abe6_43a0_8fe6_22edba5ff5d4.slice/crio-9bf04f2b103dbcdd062399d26def61de1aa07ae12ab8b5ded1e878aeaa73a7d7 WatchSource:0}: Error finding container 9bf04f2b103dbcdd062399d26def61de1aa07ae12ab8b5ded1e878aeaa73a7d7: Status 404 returned error can't find the container with id 9bf04f2b103dbcdd062399d26def61de1aa07ae12ab8b5ded1e878aeaa73a7d7
Sep 17 10:54:56.029042 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:56.028981    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-dht2n" event={"ID":"bda08aa0-553a-4c83-9f6f-fd182889e48f","Type":"ContainerStarted","Data":"347edf969bdf919b737b9398c1f02207ac707731132a75e34b2101e5150b9019"}
Sep 17 10:54:56.030834 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:56.030786    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b86b9c5f-j7sbm" event={"ID":"6e680891-abe6-43a0-8fe6-22edba5ff5d4","Type":"ContainerStarted","Data":"9bf04f2b103dbcdd062399d26def61de1aa07ae12ab8b5ded1e878aeaa73a7d7"}
Sep 17 10:54:56.032980 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:56.032952    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-vd5xp" event={"ID":"91f0be6e-7a30-4151-a73a-9c0af5212dc9","Type":"ContainerStarted","Data":"d90f132aa7fd195b2b189ea291938851fc7a4d34338170fd9bf8a2835ab362db"}
Sep 17 10:54:56.033315 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:56.033295    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-vd5xp"
Sep 17 10:54:56.035320 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:56.035299    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt" event={"ID":"98d9144c-4704-42e8-b203-dcf14e23974a","Type":"ContainerStarted","Data":"91b79d0d15ede4f3f862641800e60de7c30c74a1d8a9629ee55bf2d2e3c2bddb"}
Sep 17 10:54:56.037281 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:56.037254    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw" event={"ID":"126afa14-cfb2-47cc-b856-23679e74bf47","Type":"ContainerStarted","Data":"117c63a9dc635250a4137e787a7759212ba2493d31433583730481472c6b19e7"}
Sep 17 10:54:56.051735 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:56.051699    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-vd5xp"
Sep 17 10:54:56.055423 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:56.055364    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-vd5xp" podStartSLOduration=1.360880166 podStartE2EDuration="18.055342687s" totalImagesPullingTime="16.694462521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:54:56.052995926 +0000 UTC m=+90.151041493" watchObservedRunningTime="2026-09-17 10:54:56.055342687 +0000 UTC m=+90.153388305"
Sep 17 10:54:57.048996 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:57.048958    2562 generic.go:356] "Generic (PLEG): container finished" podID="266ba384-8031-4300-b02f-9331659c09e2" containerID="f1826a3f33f87eddcb9b2ec018026ae33445d1878a9fbe3ac94209e6c93851f5" exitCode=0
Sep 17 10:54:57.050327 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:57.050229    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"266ba384-8031-4300-b02f-9331659c09e2","Type":"ContainerDied","Data":"f1826a3f33f87eddcb9b2ec018026ae33445d1878a9fbe3ac94209e6c93851f5"}
Sep 17 10:54:57.294735 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:57.294622    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:57.363843 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:57.363742    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:59.292926 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:59.292796    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:54:59.370132 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:54:59.370079    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:00.554332 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:00.554292    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8fd5d4c56-t84nw"]
Sep 17 10:55:00.666043 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:00.665970    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:00.736591 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:00.736543    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:02.221971 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:02.221902    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:02.332901 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:02.332873    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:03.223082 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:03.223007    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:03.358933 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:03.358887    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:03.469787 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:03.469289    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:03.658727 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:03.658657    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:03.788790 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:03.788759    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:04.096529 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:04.096442    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt" event={"ID":"98d9144c-4704-42e8-b203-dcf14e23974a","Type":"ContainerStarted","Data":"75b5e567a729b9c208d4a56a115b3191e6a9ad7203ec21a601a80d0dda3b1172"}
Sep 17 10:55:04.096823 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:04.096791    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt" event={"ID":"98d9144c-4704-42e8-b203-dcf14e23974a","Type":"ContainerStarted","Data":"783e1d22025b252ba8a26ce2ee91a8035bc611e9d98e507f91eaaa6f8fcc9194"}
Sep 17 10:55:04.096936 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:04.096869    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt" event={"ID":"98d9144c-4704-42e8-b203-dcf14e23974a","Type":"ContainerStarted","Data":"be2215b69dd4f17bb82a8531eb69ff8c045ab87f98759757d3571d46f9a19975"}
Sep 17 10:55:04.098957 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:04.098932    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw" event={"ID":"126afa14-cfb2-47cc-b856-23679e74bf47","Type":"ContainerStarted","Data":"35ed277a3a6eb53ab300f7b10bb5e6d361bc24636a5ccafc61f399fbcf5be5ca"}
Sep 17 10:55:04.099055 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:04.099000    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw" event={"ID":"126afa14-cfb2-47cc-b856-23679e74bf47","Type":"ContainerStarted","Data":"97e2285a72e109ae3eab812d84ddffd4009d490ba4fc286629bda35e6b99695f"}
Sep 17 10:55:04.099055 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:04.099043    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw" event={"ID":"126afa14-cfb2-47cc-b856-23679e74bf47","Type":"ContainerStarted","Data":"a261f406df17c317733a35b4cec86ebb89b022d1b35c6fe11aa55ec1aa43a461"}
Sep 17 10:55:04.102542 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:04.102517    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"266ba384-8031-4300-b02f-9331659c09e2","Type":"ContainerStarted","Data":"6311c0dcc128bf4e70ee2fd82e68dd5e1150eaf28bc7b0186cc9aebaa27a3743"}
Sep 17 10:55:04.102641 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:04.102585    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"266ba384-8031-4300-b02f-9331659c09e2","Type":"ContainerStarted","Data":"861b8659d7c66cb2956d674e0c17100720a566f8b04892efd93bba7d4e91f255"}
Sep 17 10:55:04.102641 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:04.102625    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"266ba384-8031-4300-b02f-9331659c09e2","Type":"ContainerStarted","Data":"d6028415c0100f2232fd224eadd0293fd096463749b87fa5630ef04b7d322d01"}
Sep 17 10:55:04.102721 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:04.102655    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"266ba384-8031-4300-b02f-9331659c09e2","Type":"ContainerStarted","Data":"37b6892813afb80ff19a2e64d29456c1f53b2ea20db65ad239ad635ea73d019e"}
Sep 17 10:55:04.102721 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:04.102679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"266ba384-8031-4300-b02f-9331659c09e2","Type":"ContainerStarted","Data":"58615966b434d11ae9af57fa73d7711885598cd37ea02e87eee1ab3c3850b9f2"}
Sep 17 10:55:04.104660 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:04.104635    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-dht2n" event={"ID":"bda08aa0-553a-4c83-9f6f-fd182889e48f","Type":"ContainerStarted","Data":"031d0f803de1fc181cf8abb1a97b8ccda642d7c30d4a1a259f8a6b0e8cb8528b"}
Sep 17 10:55:04.104809 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:04.104758    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-dht2n"
Sep 17 10:55:04.106769 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:04.106718    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b86b9c5f-j7sbm" event={"ID":"6e680891-abe6-43a0-8fe6-22edba5ff5d4","Type":"ContainerStarted","Data":"1f45f4cd5668515e4384e9c75b0e17ccb7ddc7505e67eaa009d42d1835ac61b8"}
Sep 17 10:55:04.111201 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:04.111179    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-dht2n"
Sep 17 10:55:04.173428 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:04.173372    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b86b9c5f-j7sbm" podStartSLOduration=8.748966511999999 podStartE2EDuration="16.173349626s" totalImagesPullingTime="7.424383114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:55:04.171858402 +0000 UTC m=+98.269903965" watchObservedRunningTime="2026-09-17 10:55:04.173349626 +0000 UTC m=+98.271395195"
Sep 17 10:55:04.173914 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:04.173870    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-58c97c79fd-zzkqt" podStartSLOduration=5.753582196 podStartE2EDuration="13.173862764s" totalImagesPullingTime="7.420280568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:55:04.133245114 +0000 UTC m=+98.231290682" watchObservedRunningTime="2026-09-17 10:55:04.173862764 +0000 UTC m=+98.271908323"
Sep 17 10:55:04.198542 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:04.198484    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-dht2n" podStartSLOduration=6.764037507 podStartE2EDuration="14.198468594s" totalImagesPullingTime="7.434431087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:55:04.196621488 +0000 UTC m=+98.294667057" watchObservedRunningTime="2026-09-17 10:55:04.198468594 +0000 UTC m=+98.296514160"
Sep 17 10:55:06.118362 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:06.118322    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"266ba384-8031-4300-b02f-9331659c09e2","Type":"ContainerStarted","Data":"08ac1faeb27455229cee6d745b7f897f1bfd58570bcc832a71fe9eb325a18b42"}
Sep 17 10:55:06.121471 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:06.121443    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw" event={"ID":"126afa14-cfb2-47cc-b856-23679e74bf47","Type":"ContainerStarted","Data":"122b3e61737561a68248b7850a58e8665d785e3d0cdfb8b0237536bde4906fea"}
Sep 17 10:55:06.121571 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:06.121517    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw" event={"ID":"126afa14-cfb2-47cc-b856-23679e74bf47","Type":"ContainerStarted","Data":"aeb1d55ece5e6329320ce551e7e91ecc3e2ea28bab72f76776b453a6665e54e3"}
Sep 17 10:55:06.121571 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:06.121547    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw" event={"ID":"126afa14-cfb2-47cc-b856-23679e74bf47","Type":"ContainerStarted","Data":"08251a1f3759a3a1b3b477cbe853991bb7cacf5f70fb4cd84a29cdece34b25af"}
Sep 17 10:55:06.176530 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:06.176459    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.45812941 podStartE2EDuration="19.176439891s" totalImagesPullingTime="15.698284597s" totalInitContainerRuntime="20.025884ms" isStatefulPod=false podCreationTimestamp="2026-09-17 10:54:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:55:06.175133307 +0000 UTC m=+100.273178877" watchObservedRunningTime="2026-09-17 10:55:06.176439891 +0000 UTC m=+100.274485459"
Sep 17 10:55:06.233002 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:06.232776    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw" podStartSLOduration=8.045545402 podStartE2EDuration="17.232752549s" totalImagesPullingTime="9.187207147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:54:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:55:06.231668211 +0000 UTC m=+100.329713800" watchObservedRunningTime="2026-09-17 10:55:06.232752549 +0000 UTC m=+100.330798117"
Sep 17 10:55:07.126517 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:07.126479    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:55:07.451289 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:07.451206    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b86b9c5f-j7sbm"]
Sep 17 10:55:07.464521 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:07.464492    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:07.659440 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:07.659409    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:07.780344 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:07.780294    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:08.136192 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:08.136117    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7fc69f886-q7tdw"
Sep 17 10:55:08.616409 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:08.616372    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:08.680894 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:08.680857    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:55:08.777828 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:08.777776    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:12.101198 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:12.101154    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:12.179672 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:12.179640    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:18.836054 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:18.836017    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:18.937170 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:18.937136    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:19.005615 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:19.005569    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:22.541094 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:22.541061    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:22.642642 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:22.642614    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:55:25.201286 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.201254    2562 generic.go:356] "Generic (PLEG): container finished" podID="3a72b5be-ac50-4f83-89ab-f0664cf51f1f" containerID="7fe60b279d2a422eb6e2e7ec7e507844a43b5ff35973ca38c810d92f4a56e71e" exitCode=0
Sep 17 10:55:25.201651 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.201327    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-dpdf4" event={"ID":"3a72b5be-ac50-4f83-89ab-f0664cf51f1f","Type":"ContainerDied","Data":"7fe60b279d2a422eb6e2e7ec7e507844a43b5ff35973ca38c810d92f4a56e71e"}
Sep 17 10:55:25.201651 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.201642    2562 scope.go:117] "RemoveContainer" containerID="7fe60b279d2a422eb6e2e7ec7e507844a43b5ff35973ca38c810d92f4a56e71e"
Sep 17 10:55:25.578176 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.578118    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw" podUID="97de9f7d-09e9-4412-9c7e-1eb4f2a4d447" containerName="registry" containerID="cri-o://23d6f19da8b14cade1a1b83add243dccfeed5106ae16e52d604c69753353015e" gracePeriod=30
Sep 17 10:55:25.824835 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.824793    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:55:25.880566 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.880480    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-ca-trust-extracted\") pod \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") "
Sep 17 10:55:25.880566 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.880535    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-trusted-ca\") pod \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") "
Sep 17 10:55:25.880566 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.880563    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-certificates\") pod \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") "
Sep 17 10:55:25.880861 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.880581    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-image-registry-private-configuration\") pod \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") "
Sep 17 10:55:25.880861 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.880604    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-installation-pull-secrets\") pod \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") "
Sep 17 10:55:25.880861 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.880623    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-bound-sa-token\") pod \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") "
Sep 17 10:55:25.880861 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.880640    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls\") pod \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") "
Sep 17 10:55:25.880861 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.880709    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmwgm\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-kube-api-access-qmwgm\") pod \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\" (UID: \"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447\") "
Sep 17 10:55:25.881119 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.881034    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447" (UID: "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 10:55:25.881614 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.881563    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447" (UID: "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 10:55:25.884443 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.884386    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-kube-api-access-qmwgm" (OuterVolumeSpecName: "kube-api-access-qmwgm") pod "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447" (UID: "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447"). InnerVolumeSpecName "kube-api-access-qmwgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 10:55:25.884443 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.884386    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447" (UID: "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 10:55:25.884625 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.884493    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447" (UID: "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 10:55:25.884686 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.884669    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447" (UID: "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 10:55:25.884794 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.884776    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447" (UID: "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 10:55:25.890652 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.890628    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447" (UID: "97de9f7d-09e9-4412-9c7e-1eb4f2a4d447"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 10:55:25.981577 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.981537    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-ca-trust-extracted\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:55:25.981577 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.981569    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-trusted-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:55:25.981577 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.981579    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-certificates\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:55:25.981787 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.981591    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-image-registry-private-configuration\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:55:25.981787 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.981601    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-installation-pull-secrets\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:55:25.981787 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.981609    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-bound-sa-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:55:25.981787 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.981617    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-registry-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:55:25.981787 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:25.981625    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmwgm\" (UniqueName: \"kubernetes.io/projected/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447-kube-api-access-qmwgm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:55:26.207997 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:26.207892    2562 generic.go:356] "Generic (PLEG): container finished" podID="97de9f7d-09e9-4412-9c7e-1eb4f2a4d447" containerID="23d6f19da8b14cade1a1b83add243dccfeed5106ae16e52d604c69753353015e" exitCode=0
Sep 17 10:55:26.207997 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:26.207945    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw"
Sep 17 10:55:26.207997 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:26.207978    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw" event={"ID":"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447","Type":"ContainerDied","Data":"23d6f19da8b14cade1a1b83add243dccfeed5106ae16e52d604c69753353015e"}
Sep 17 10:55:26.208630 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:26.208068    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8fd5d4c56-t84nw" event={"ID":"97de9f7d-09e9-4412-9c7e-1eb4f2a4d447","Type":"ContainerDied","Data":"13407a0e57fe703f6c89a60607e247eb9e433762d3e09a0dca87ea4dd22b10d6"}
Sep 17 10:55:26.208630 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:26.208101    2562 scope.go:117] "RemoveContainer" containerID="23d6f19da8b14cade1a1b83add243dccfeed5106ae16e52d604c69753353015e"
Sep 17 10:55:26.210492 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:26.210461    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-dpdf4" event={"ID":"3a72b5be-ac50-4f83-89ab-f0664cf51f1f","Type":"ContainerStarted","Data":"31533e74d097db06f30bde06e11e53ab08328f749d7590bb38241077e9ae6c23"}
Sep 17 10:55:26.218461 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:26.218442    2562 scope.go:117] "RemoveContainer" containerID="23d6f19da8b14cade1a1b83add243dccfeed5106ae16e52d604c69753353015e"
Sep 17 10:55:26.218784 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:55:26.218761    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d6f19da8b14cade1a1b83add243dccfeed5106ae16e52d604c69753353015e\": container with ID starting with 23d6f19da8b14cade1a1b83add243dccfeed5106ae16e52d604c69753353015e not found: ID does not exist" containerID="23d6f19da8b14cade1a1b83add243dccfeed5106ae16e52d604c69753353015e"
Sep 17 10:55:26.218874 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:26.218793    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d6f19da8b14cade1a1b83add243dccfeed5106ae16e52d604c69753353015e"} err="failed to get container status \"23d6f19da8b14cade1a1b83add243dccfeed5106ae16e52d604c69753353015e\": rpc error: code = NotFound desc = could not find container \"23d6f19da8b14cade1a1b83add243dccfeed5106ae16e52d604c69753353015e\": container with ID starting with 23d6f19da8b14cade1a1b83add243dccfeed5106ae16e52d604c69753353015e not found: ID does not exist"
Sep 17 10:55:26.251723 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:26.251689    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8fd5d4c56-t84nw"]
Sep 17 10:55:26.257126 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:26.257095    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8fd5d4c56-t84nw"]
Sep 17 10:55:26.455277 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:26.455239    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97de9f7d-09e9-4412-9c7e-1eb4f2a4d447" path="/var/lib/kubelet/pods/97de9f7d-09e9-4412-9c7e-1eb4f2a4d447/volumes"
Sep 17 10:55:30.229135 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:30.229100    2562 generic.go:356] "Generic (PLEG): container finished" podID="8d1fa8ec-467e-4787-b213-86b104ae0e61" containerID="abecc223c6a6331862538819d3756786c563b31a83ea9f3821374d081f67449b" exitCode=0
Sep 17 10:55:30.229585 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:30.229171    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t" event={"ID":"8d1fa8ec-467e-4787-b213-86b104ae0e61","Type":"ContainerDied","Data":"abecc223c6a6331862538819d3756786c563b31a83ea9f3821374d081f67449b"}
Sep 17 10:55:30.229585 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:30.229497    2562 scope.go:117] "RemoveContainer" containerID="abecc223c6a6331862538819d3756786c563b31a83ea9f3821374d081f67449b"
Sep 17 10:55:31.235271 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:31.235232    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z6w7t" event={"ID":"8d1fa8ec-467e-4787-b213-86b104ae0e61","Type":"ContainerStarted","Data":"9358f4f2cb9e460c8ee52992671b0001ebd605007082b72c047b9202ae6f6352"}
Sep 17 10:55:32.470850 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.470778    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b86b9c5f-j7sbm" podUID="6e680891-abe6-43a0-8fe6-22edba5ff5d4" containerName="console" containerID="cri-o://1f45f4cd5668515e4384e9c75b0e17ccb7ddc7505e67eaa009d42d1835ac61b8" gracePeriod=15
Sep 17 10:55:32.757105 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.757069    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:55:32.842497 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.842456    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e680891-abe6-43a0-8fe6-22edba5ff5d4-console-config\") pod \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") "
Sep 17 10:55:32.842497 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.842501    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e680891-abe6-43a0-8fe6-22edba5ff5d4-console-serving-cert\") pod \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") "
Sep 17 10:55:32.842750 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.842522    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e680891-abe6-43a0-8fe6-22edba5ff5d4-console-oauth-config\") pod \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") "
Sep 17 10:55:32.842750 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.842554    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e680891-abe6-43a0-8fe6-22edba5ff5d4-oauth-serving-cert\") pod \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") "
Sep 17 10:55:32.842750 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.842588    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgrm9\" (UniqueName: \"kubernetes.io/projected/6e680891-abe6-43a0-8fe6-22edba5ff5d4-kube-api-access-lgrm9\") pod \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") "
Sep 17 10:55:32.842750 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.842651    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e680891-abe6-43a0-8fe6-22edba5ff5d4-service-ca\") pod \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\" (UID: \"6e680891-abe6-43a0-8fe6-22edba5ff5d4\") "
Sep 17 10:55:32.843031 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.843004    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e680891-abe6-43a0-8fe6-22edba5ff5d4-console-config" (OuterVolumeSpecName: "console-config") pod "6e680891-abe6-43a0-8fe6-22edba5ff5d4" (UID: "6e680891-abe6-43a0-8fe6-22edba5ff5d4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 10:55:32.843132 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.843097    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e680891-abe6-43a0-8fe6-22edba5ff5d4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6e680891-abe6-43a0-8fe6-22edba5ff5d4" (UID: "6e680891-abe6-43a0-8fe6-22edba5ff5d4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 10:55:32.843512 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.843493    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e680891-abe6-43a0-8fe6-22edba5ff5d4-console-config\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:55:32.843575 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.843517    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e680891-abe6-43a0-8fe6-22edba5ff5d4-oauth-serving-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:55:32.843748 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.843717    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e680891-abe6-43a0-8fe6-22edba5ff5d4-service-ca" (OuterVolumeSpecName: "service-ca") pod "6e680891-abe6-43a0-8fe6-22edba5ff5d4" (UID: "6e680891-abe6-43a0-8fe6-22edba5ff5d4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 10:55:32.845342 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.845316    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e680891-abe6-43a0-8fe6-22edba5ff5d4-kube-api-access-lgrm9" (OuterVolumeSpecName: "kube-api-access-lgrm9") pod "6e680891-abe6-43a0-8fe6-22edba5ff5d4" (UID: "6e680891-abe6-43a0-8fe6-22edba5ff5d4"). InnerVolumeSpecName "kube-api-access-lgrm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 10:55:32.845436 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.845417    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e680891-abe6-43a0-8fe6-22edba5ff5d4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6e680891-abe6-43a0-8fe6-22edba5ff5d4" (UID: "6e680891-abe6-43a0-8fe6-22edba5ff5d4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 10:55:32.845477 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.845434    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e680891-abe6-43a0-8fe6-22edba5ff5d4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6e680891-abe6-43a0-8fe6-22edba5ff5d4" (UID: "6e680891-abe6-43a0-8fe6-22edba5ff5d4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 10:55:32.944348 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.944308    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e680891-abe6-43a0-8fe6-22edba5ff5d4-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:55:32.944348 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.944343    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e680891-abe6-43a0-8fe6-22edba5ff5d4-console-serving-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:55:32.944348 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.944356    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e680891-abe6-43a0-8fe6-22edba5ff5d4-console-oauth-config\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:55:32.944577 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:32.944370    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgrm9\" (UniqueName: \"kubernetes.io/projected/6e680891-abe6-43a0-8fe6-22edba5ff5d4-kube-api-access-lgrm9\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:55:33.245599 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:33.245565    2562 generic.go:356] "Generic (PLEG): container finished" podID="6e680891-abe6-43a0-8fe6-22edba5ff5d4" containerID="1f45f4cd5668515e4384e9c75b0e17ccb7ddc7505e67eaa009d42d1835ac61b8" exitCode=0
Sep 17 10:55:33.245781 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:33.245629    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b86b9c5f-j7sbm"
Sep 17 10:55:33.245781 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:33.245634    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b86b9c5f-j7sbm" event={"ID":"6e680891-abe6-43a0-8fe6-22edba5ff5d4","Type":"ContainerDied","Data":"1f45f4cd5668515e4384e9c75b0e17ccb7ddc7505e67eaa009d42d1835ac61b8"}
Sep 17 10:55:33.245781 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:33.245744    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b86b9c5f-j7sbm" event={"ID":"6e680891-abe6-43a0-8fe6-22edba5ff5d4","Type":"ContainerDied","Data":"9bf04f2b103dbcdd062399d26def61de1aa07ae12ab8b5ded1e878aeaa73a7d7"}
Sep 17 10:55:33.245781 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:33.245769    2562 scope.go:117] "RemoveContainer" containerID="1f45f4cd5668515e4384e9c75b0e17ccb7ddc7505e67eaa009d42d1835ac61b8"
Sep 17 10:55:33.261960 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:33.261931    2562 scope.go:117] "RemoveContainer" containerID="1f45f4cd5668515e4384e9c75b0e17ccb7ddc7505e67eaa009d42d1835ac61b8"
Sep 17 10:55:33.262423 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:55:33.262399    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f45f4cd5668515e4384e9c75b0e17ccb7ddc7505e67eaa009d42d1835ac61b8\": container with ID starting with 1f45f4cd5668515e4384e9c75b0e17ccb7ddc7505e67eaa009d42d1835ac61b8 not found: ID does not exist" containerID="1f45f4cd5668515e4384e9c75b0e17ccb7ddc7505e67eaa009d42d1835ac61b8"
Sep 17 10:55:33.262519 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:33.262429    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f45f4cd5668515e4384e9c75b0e17ccb7ddc7505e67eaa009d42d1835ac61b8"} err="failed to get container status \"1f45f4cd5668515e4384e9c75b0e17ccb7ddc7505e67eaa009d42d1835ac61b8\": rpc error: code = NotFound desc = could not find container \"1f45f4cd5668515e4384e9c75b0e17ccb7ddc7505e67eaa009d42d1835ac61b8\": container with ID starting with 1f45f4cd5668515e4384e9c75b0e17ccb7ddc7505e67eaa009d42d1835ac61b8 not found: ID does not exist"
Sep 17 10:55:33.275396 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:33.275362    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b86b9c5f-j7sbm"]
Sep 17 10:55:33.279377 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:33.279353    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b86b9c5f-j7sbm"]
Sep 17 10:55:34.454967 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:34.454913    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e680891-abe6-43a0-8fe6-22edba5ff5d4" path="/var/lib/kubelet/pods/6e680891-abe6-43a0-8fe6-22edba5ff5d4/volumes"
Sep 17 10:55:35.256634 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:35.256598    2562 generic.go:356] "Generic (PLEG): container finished" podID="ad3b878d-f419-4e39-88d1-a53ee931a22e" containerID="f450038898832301540d1cb284ac408074c104e209e109a4ddec1b91b2d52278" exitCode=0
Sep 17 10:55:35.256831 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:35.256674    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r" event={"ID":"ad3b878d-f419-4e39-88d1-a53ee931a22e","Type":"ContainerDied","Data":"f450038898832301540d1cb284ac408074c104e209e109a4ddec1b91b2d52278"}
Sep 17 10:55:35.257049 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:35.257034    2562 scope.go:117] "RemoveContainer" containerID="f450038898832301540d1cb284ac408074c104e209e109a4ddec1b91b2d52278"
Sep 17 10:55:36.263835 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:55:36.263789    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mrd5r" event={"ID":"ad3b878d-f419-4e39-88d1-a53ee931a22e","Type":"ContainerStarted","Data":"a521c5aac9033240e7f115af4497b5d63e2e5c97311c1cdc8d2facbbdc5fcd54"}
Sep 17 10:56:02.474239 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:02.474202    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:02.564343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:02.564309    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:06.786682 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:06.786644    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 10:56:06.787149 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:06.786925    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="alertmanager" containerID="cri-o://58615966b434d11ae9af57fa73d7711885598cd37ea02e87eee1ab3c3850b9f2" gracePeriod=120
Sep 17 10:56:06.787149 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:06.786988    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="kube-rbac-proxy-metric" containerID="cri-o://6311c0dcc128bf4e70ee2fd82e68dd5e1150eaf28bc7b0186cc9aebaa27a3743" gracePeriod=120
Sep 17 10:56:06.787149 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:06.787008    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="config-reloader" containerID="cri-o://37b6892813afb80ff19a2e64d29456c1f53b2ea20db65ad239ad635ea73d019e" gracePeriod=120
Sep 17 10:56:06.787149 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:06.787023    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="kube-rbac-proxy" containerID="cri-o://861b8659d7c66cb2956d674e0c17100720a566f8b04892efd93bba7d4e91f255" gracePeriod=120
Sep 17 10:56:06.787149 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:06.787053    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="prom-label-proxy" containerID="cri-o://08ac1faeb27455229cee6d745b7f897f1bfd58570bcc832a71fe9eb325a18b42" gracePeriod=120
Sep 17 10:56:06.787149 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:06.787000    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="kube-rbac-proxy-web" containerID="cri-o://d6028415c0100f2232fd224eadd0293fd096463749b87fa5630ef04b7d322d01" gracePeriod=120
Sep 17 10:56:07.405261 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:07.405224    2562 generic.go:356] "Generic (PLEG): container finished" podID="266ba384-8031-4300-b02f-9331659c09e2" containerID="08ac1faeb27455229cee6d745b7f897f1bfd58570bcc832a71fe9eb325a18b42" exitCode=0
Sep 17 10:56:07.405261 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:07.405251    2562 generic.go:356] "Generic (PLEG): container finished" podID="266ba384-8031-4300-b02f-9331659c09e2" containerID="6311c0dcc128bf4e70ee2fd82e68dd5e1150eaf28bc7b0186cc9aebaa27a3743" exitCode=0
Sep 17 10:56:07.405261 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:07.405257    2562 generic.go:356] "Generic (PLEG): container finished" podID="266ba384-8031-4300-b02f-9331659c09e2" containerID="861b8659d7c66cb2956d674e0c17100720a566f8b04892efd93bba7d4e91f255" exitCode=0
Sep 17 10:56:07.405261 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:07.405263    2562 generic.go:356] "Generic (PLEG): container finished" podID="266ba384-8031-4300-b02f-9331659c09e2" containerID="37b6892813afb80ff19a2e64d29456c1f53b2ea20db65ad239ad635ea73d019e" exitCode=0
Sep 17 10:56:07.405261 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:07.405269    2562 generic.go:356] "Generic (PLEG): container finished" podID="266ba384-8031-4300-b02f-9331659c09e2" containerID="58615966b434d11ae9af57fa73d7711885598cd37ea02e87eee1ab3c3850b9f2" exitCode=0
Sep 17 10:56:07.405545 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:07.405292    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"266ba384-8031-4300-b02f-9331659c09e2","Type":"ContainerDied","Data":"08ac1faeb27455229cee6d745b7f897f1bfd58570bcc832a71fe9eb325a18b42"}
Sep 17 10:56:07.405545 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:07.405355    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"266ba384-8031-4300-b02f-9331659c09e2","Type":"ContainerDied","Data":"6311c0dcc128bf4e70ee2fd82e68dd5e1150eaf28bc7b0186cc9aebaa27a3743"}
Sep 17 10:56:07.405545 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:07.405384    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"266ba384-8031-4300-b02f-9331659c09e2","Type":"ContainerDied","Data":"861b8659d7c66cb2956d674e0c17100720a566f8b04892efd93bba7d4e91f255"}
Sep 17 10:56:07.405545 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:07.405408    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"266ba384-8031-4300-b02f-9331659c09e2","Type":"ContainerDied","Data":"37b6892813afb80ff19a2e64d29456c1f53b2ea20db65ad239ad635ea73d019e"}
Sep 17 10:56:07.405545 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:07.405431    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"266ba384-8031-4300-b02f-9331659c09e2","Type":"ContainerDied","Data":"58615966b434d11ae9af57fa73d7711885598cd37ea02e87eee1ab3c3850b9f2"}
Sep 17 10:56:08.040216 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.040185    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.121550 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.121481    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"266ba384-8031-4300-b02f-9331659c09e2\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") "
Sep 17 10:56:08.121692 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.121558    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/266ba384-8031-4300-b02f-9331659c09e2-metrics-client-ca\") pod \"266ba384-8031-4300-b02f-9331659c09e2\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") "
Sep 17 10:56:08.121692 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.121580    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-config-volume\") pod \"266ba384-8031-4300-b02f-9331659c09e2\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") "
Sep 17 10:56:08.121692 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.121599    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-main-tls\") pod \"266ba384-8031-4300-b02f-9331659c09e2\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") "
Sep 17 10:56:08.121692 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.121620    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/266ba384-8031-4300-b02f-9331659c09e2-config-out\") pod \"266ba384-8031-4300-b02f-9331659c09e2\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") "
Sep 17 10:56:08.121692 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.121638    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-cluster-tls-config\") pod \"266ba384-8031-4300-b02f-9331659c09e2\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") "
Sep 17 10:56:08.121692 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.121685    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-web-config\") pod \"266ba384-8031-4300-b02f-9331659c09e2\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") "
Sep 17 10:56:08.122081 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.121719    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4pjq\" (UniqueName: \"kubernetes.io/projected/266ba384-8031-4300-b02f-9331659c09e2-kube-api-access-n4pjq\") pod \"266ba384-8031-4300-b02f-9331659c09e2\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") "
Sep 17 10:56:08.122081 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.121827    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/266ba384-8031-4300-b02f-9331659c09e2-tls-assets\") pod \"266ba384-8031-4300-b02f-9331659c09e2\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") "
Sep 17 10:56:08.122081 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.121881    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/266ba384-8031-4300-b02f-9331659c09e2-alertmanager-main-db\") pod \"266ba384-8031-4300-b02f-9331659c09e2\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") "
Sep 17 10:56:08.122081 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.121913    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-kube-rbac-proxy\") pod \"266ba384-8031-4300-b02f-9331659c09e2\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") "
Sep 17 10:56:08.122081 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.121938    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/266ba384-8031-4300-b02f-9331659c09e2-alertmanager-trusted-ca-bundle\") pod \"266ba384-8031-4300-b02f-9331659c09e2\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") "
Sep 17 10:56:08.122081 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.121966    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"266ba384-8031-4300-b02f-9331659c09e2\" (UID: \"266ba384-8031-4300-b02f-9331659c09e2\") "
Sep 17 10:56:08.124882 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.124835    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "266ba384-8031-4300-b02f-9331659c09e2" (UID: "266ba384-8031-4300-b02f-9331659c09e2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 10:56:08.124882 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.124856    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266ba384-8031-4300-b02f-9331659c09e2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "266ba384-8031-4300-b02f-9331659c09e2" (UID: "266ba384-8031-4300-b02f-9331659c09e2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 10:56:08.125441 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.125214    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "266ba384-8031-4300-b02f-9331659c09e2" (UID: "266ba384-8031-4300-b02f-9331659c09e2"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 10:56:08.125441 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.125427    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "266ba384-8031-4300-b02f-9331659c09e2" (UID: "266ba384-8031-4300-b02f-9331659c09e2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 10:56:08.125779 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.125743    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266ba384-8031-4300-b02f-9331659c09e2-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "266ba384-8031-4300-b02f-9331659c09e2" (UID: "266ba384-8031-4300-b02f-9331659c09e2"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 10:56:08.125779 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.125758    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266ba384-8031-4300-b02f-9331659c09e2-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "266ba384-8031-4300-b02f-9331659c09e2" (UID: "266ba384-8031-4300-b02f-9331659c09e2"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 10:56:08.125951 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.125772    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266ba384-8031-4300-b02f-9331659c09e2-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "266ba384-8031-4300-b02f-9331659c09e2" (UID: "266ba384-8031-4300-b02f-9331659c09e2"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 10:56:08.126038 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.126012    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266ba384-8031-4300-b02f-9331659c09e2-config-out" (OuterVolumeSpecName: "config-out") pod "266ba384-8031-4300-b02f-9331659c09e2" (UID: "266ba384-8031-4300-b02f-9331659c09e2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 10:56:08.126497 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.126465    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-config-volume" (OuterVolumeSpecName: "config-volume") pod "266ba384-8031-4300-b02f-9331659c09e2" (UID: "266ba384-8031-4300-b02f-9331659c09e2"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 10:56:08.127123 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.127101    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "266ba384-8031-4300-b02f-9331659c09e2" (UID: "266ba384-8031-4300-b02f-9331659c09e2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 10:56:08.127277 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.127252    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266ba384-8031-4300-b02f-9331659c09e2-kube-api-access-n4pjq" (OuterVolumeSpecName: "kube-api-access-n4pjq") pod "266ba384-8031-4300-b02f-9331659c09e2" (UID: "266ba384-8031-4300-b02f-9331659c09e2"). InnerVolumeSpecName "kube-api-access-n4pjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 10:56:08.131354 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.131176    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "266ba384-8031-4300-b02f-9331659c09e2" (UID: "266ba384-8031-4300-b02f-9331659c09e2"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 10:56:08.138384 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.138354    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-web-config" (OuterVolumeSpecName: "web-config") pod "266ba384-8031-4300-b02f-9331659c09e2" (UID: "266ba384-8031-4300-b02f-9331659c09e2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 10:56:08.223002 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.222946    2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:56:08.223002 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.222995    2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/266ba384-8031-4300-b02f-9331659c09e2-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:56:08.223002 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.223006    2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:56:08.223002 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.223015    2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:56:08.223275 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.223024    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/266ba384-8031-4300-b02f-9331659c09e2-metrics-client-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:56:08.223275 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.223034    2562 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:56:08.223275 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.223042    2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-secret-alertmanager-main-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:56:08.223275 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.223049    2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/266ba384-8031-4300-b02f-9331659c09e2-config-out\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:56:08.223275 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.223057    2562 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-cluster-tls-config\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:56:08.223275 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.223065    2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/266ba384-8031-4300-b02f-9331659c09e2-web-config\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:56:08.223275 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.223073    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4pjq\" (UniqueName: \"kubernetes.io/projected/266ba384-8031-4300-b02f-9331659c09e2-kube-api-access-n4pjq\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:56:08.223275 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.223082    2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/266ba384-8031-4300-b02f-9331659c09e2-tls-assets\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:56:08.223275 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.223091    2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/266ba384-8031-4300-b02f-9331659c09e2-alertmanager-main-db\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 10:56:08.411954 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.411870    2562 generic.go:356] "Generic (PLEG): container finished" podID="266ba384-8031-4300-b02f-9331659c09e2" containerID="d6028415c0100f2232fd224eadd0293fd096463749b87fa5630ef04b7d322d01" exitCode=0
Sep 17 10:56:08.411954 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.411920    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"266ba384-8031-4300-b02f-9331659c09e2","Type":"ContainerDied","Data":"d6028415c0100f2232fd224eadd0293fd096463749b87fa5630ef04b7d322d01"}
Sep 17 10:56:08.411954 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.411943    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.412178 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.411988    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"266ba384-8031-4300-b02f-9331659c09e2","Type":"ContainerDied","Data":"435563c1f06c3c1de7ef45714c382f4294e6280aecfd12509d8cc1a73c6a5e76"}
Sep 17 10:56:08.412178 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.412020    2562 scope.go:117] "RemoveContainer" containerID="08ac1faeb27455229cee6d745b7f897f1bfd58570bcc832a71fe9eb325a18b42"
Sep 17 10:56:08.421020 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.421002    2562 scope.go:117] "RemoveContainer" containerID="6311c0dcc128bf4e70ee2fd82e68dd5e1150eaf28bc7b0186cc9aebaa27a3743"
Sep 17 10:56:08.428219 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.428200    2562 scope.go:117] "RemoveContainer" containerID="861b8659d7c66cb2956d674e0c17100720a566f8b04892efd93bba7d4e91f255"
Sep 17 10:56:08.434978 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.434960    2562 scope.go:117] "RemoveContainer" containerID="d6028415c0100f2232fd224eadd0293fd096463749b87fa5630ef04b7d322d01"
Sep 17 10:56:08.437943 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.437919    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 10:56:08.442665 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.442640    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 10:56:08.443244 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.443224    2562 scope.go:117] "RemoveContainer" containerID="37b6892813afb80ff19a2e64d29456c1f53b2ea20db65ad239ad635ea73d019e"
Sep 17 10:56:08.455651 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.451189    2562 scope.go:117] "RemoveContainer" containerID="58615966b434d11ae9af57fa73d7711885598cd37ea02e87eee1ab3c3850b9f2"
Sep 17 10:56:08.459293 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.459270    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="266ba384-8031-4300-b02f-9331659c09e2" path="/var/lib/kubelet/pods/266ba384-8031-4300-b02f-9331659c09e2/volumes"
Sep 17 10:56:08.463213 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.463196    2562 scope.go:117] "RemoveContainer" containerID="f1826a3f33f87eddcb9b2ec018026ae33445d1878a9fbe3ac94209e6c93851f5"
Sep 17 10:56:08.470354 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.470333    2562 scope.go:117] "RemoveContainer" containerID="08ac1faeb27455229cee6d745b7f897f1bfd58570bcc832a71fe9eb325a18b42"
Sep 17 10:56:08.470638 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:56:08.470619    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ac1faeb27455229cee6d745b7f897f1bfd58570bcc832a71fe9eb325a18b42\": container with ID starting with 08ac1faeb27455229cee6d745b7f897f1bfd58570bcc832a71fe9eb325a18b42 not found: ID does not exist" containerID="08ac1faeb27455229cee6d745b7f897f1bfd58570bcc832a71fe9eb325a18b42"
Sep 17 10:56:08.470686 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.470648    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ac1faeb27455229cee6d745b7f897f1bfd58570bcc832a71fe9eb325a18b42"} err="failed to get container status \"08ac1faeb27455229cee6d745b7f897f1bfd58570bcc832a71fe9eb325a18b42\": rpc error: code = NotFound desc = could not find container \"08ac1faeb27455229cee6d745b7f897f1bfd58570bcc832a71fe9eb325a18b42\": container with ID starting with 08ac1faeb27455229cee6d745b7f897f1bfd58570bcc832a71fe9eb325a18b42 not found: ID does not exist"
Sep 17 10:56:08.470686 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.470668    2562 scope.go:117] "RemoveContainer" containerID="6311c0dcc128bf4e70ee2fd82e68dd5e1150eaf28bc7b0186cc9aebaa27a3743"
Sep 17 10:56:08.470971 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:56:08.470953    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6311c0dcc128bf4e70ee2fd82e68dd5e1150eaf28bc7b0186cc9aebaa27a3743\": container with ID starting with 6311c0dcc128bf4e70ee2fd82e68dd5e1150eaf28bc7b0186cc9aebaa27a3743 not found: ID does not exist" containerID="6311c0dcc128bf4e70ee2fd82e68dd5e1150eaf28bc7b0186cc9aebaa27a3743"
Sep 17 10:56:08.471024 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.470977    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6311c0dcc128bf4e70ee2fd82e68dd5e1150eaf28bc7b0186cc9aebaa27a3743"} err="failed to get container status \"6311c0dcc128bf4e70ee2fd82e68dd5e1150eaf28bc7b0186cc9aebaa27a3743\": rpc error: code = NotFound desc = could not find container \"6311c0dcc128bf4e70ee2fd82e68dd5e1150eaf28bc7b0186cc9aebaa27a3743\": container with ID starting with 6311c0dcc128bf4e70ee2fd82e68dd5e1150eaf28bc7b0186cc9aebaa27a3743 not found: ID does not exist"
Sep 17 10:56:08.471024 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.470992    2562 scope.go:117] "RemoveContainer" containerID="861b8659d7c66cb2956d674e0c17100720a566f8b04892efd93bba7d4e91f255"
Sep 17 10:56:08.471253 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:56:08.471230    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"861b8659d7c66cb2956d674e0c17100720a566f8b04892efd93bba7d4e91f255\": container with ID starting with 861b8659d7c66cb2956d674e0c17100720a566f8b04892efd93bba7d4e91f255 not found: ID does not exist" containerID="861b8659d7c66cb2956d674e0c17100720a566f8b04892efd93bba7d4e91f255"
Sep 17 10:56:08.471299 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.471260    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"861b8659d7c66cb2956d674e0c17100720a566f8b04892efd93bba7d4e91f255"} err="failed to get container status \"861b8659d7c66cb2956d674e0c17100720a566f8b04892efd93bba7d4e91f255\": rpc error: code = NotFound desc = could not find container \"861b8659d7c66cb2956d674e0c17100720a566f8b04892efd93bba7d4e91f255\": container with ID starting with 861b8659d7c66cb2956d674e0c17100720a566f8b04892efd93bba7d4e91f255 not found: ID does not exist"
Sep 17 10:56:08.471299 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.471277    2562 scope.go:117] "RemoveContainer" containerID="d6028415c0100f2232fd224eadd0293fd096463749b87fa5630ef04b7d322d01"
Sep 17 10:56:08.471513 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:56:08.471495    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6028415c0100f2232fd224eadd0293fd096463749b87fa5630ef04b7d322d01\": container with ID starting with d6028415c0100f2232fd224eadd0293fd096463749b87fa5630ef04b7d322d01 not found: ID does not exist" containerID="d6028415c0100f2232fd224eadd0293fd096463749b87fa5630ef04b7d322d01"
Sep 17 10:56:08.471555 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.471520    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6028415c0100f2232fd224eadd0293fd096463749b87fa5630ef04b7d322d01"} err="failed to get container status \"d6028415c0100f2232fd224eadd0293fd096463749b87fa5630ef04b7d322d01\": rpc error: code = NotFound desc = could not find container \"d6028415c0100f2232fd224eadd0293fd096463749b87fa5630ef04b7d322d01\": container with ID starting with d6028415c0100f2232fd224eadd0293fd096463749b87fa5630ef04b7d322d01 not found: ID does not exist"
Sep 17 10:56:08.471555 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.471539    2562 scope.go:117] "RemoveContainer" containerID="37b6892813afb80ff19a2e64d29456c1f53b2ea20db65ad239ad635ea73d019e"
Sep 17 10:56:08.471757 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:56:08.471741    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b6892813afb80ff19a2e64d29456c1f53b2ea20db65ad239ad635ea73d019e\": container with ID starting with 37b6892813afb80ff19a2e64d29456c1f53b2ea20db65ad239ad635ea73d019e not found: ID does not exist" containerID="37b6892813afb80ff19a2e64d29456c1f53b2ea20db65ad239ad635ea73d019e"
Sep 17 10:56:08.471804 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.471758    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b6892813afb80ff19a2e64d29456c1f53b2ea20db65ad239ad635ea73d019e"} err="failed to get container status \"37b6892813afb80ff19a2e64d29456c1f53b2ea20db65ad239ad635ea73d019e\": rpc error: code = NotFound desc = could not find container \"37b6892813afb80ff19a2e64d29456c1f53b2ea20db65ad239ad635ea73d019e\": container with ID starting with 37b6892813afb80ff19a2e64d29456c1f53b2ea20db65ad239ad635ea73d019e not found: ID does not exist"
Sep 17 10:56:08.471804 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.471770    2562 scope.go:117] "RemoveContainer" containerID="58615966b434d11ae9af57fa73d7711885598cd37ea02e87eee1ab3c3850b9f2"
Sep 17 10:56:08.471995 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:56:08.471979    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58615966b434d11ae9af57fa73d7711885598cd37ea02e87eee1ab3c3850b9f2\": container with ID starting with 58615966b434d11ae9af57fa73d7711885598cd37ea02e87eee1ab3c3850b9f2 not found: ID does not exist" containerID="58615966b434d11ae9af57fa73d7711885598cd37ea02e87eee1ab3c3850b9f2"
Sep 17 10:56:08.472034 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.471999    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58615966b434d11ae9af57fa73d7711885598cd37ea02e87eee1ab3c3850b9f2"} err="failed to get container status \"58615966b434d11ae9af57fa73d7711885598cd37ea02e87eee1ab3c3850b9f2\": rpc error: code = NotFound desc = could not find container \"58615966b434d11ae9af57fa73d7711885598cd37ea02e87eee1ab3c3850b9f2\": container with ID starting with 58615966b434d11ae9af57fa73d7711885598cd37ea02e87eee1ab3c3850b9f2 not found: ID does not exist"
Sep 17 10:56:08.472034 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.472013    2562 scope.go:117] "RemoveContainer" containerID="f1826a3f33f87eddcb9b2ec018026ae33445d1878a9fbe3ac94209e6c93851f5"
Sep 17 10:56:08.472235 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:56:08.472217    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1826a3f33f87eddcb9b2ec018026ae33445d1878a9fbe3ac94209e6c93851f5\": container with ID starting with f1826a3f33f87eddcb9b2ec018026ae33445d1878a9fbe3ac94209e6c93851f5 not found: ID does not exist" containerID="f1826a3f33f87eddcb9b2ec018026ae33445d1878a9fbe3ac94209e6c93851f5"
Sep 17 10:56:08.472284 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.472238    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1826a3f33f87eddcb9b2ec018026ae33445d1878a9fbe3ac94209e6c93851f5"} err="failed to get container status \"f1826a3f33f87eddcb9b2ec018026ae33445d1878a9fbe3ac94209e6c93851f5\": rpc error: code = NotFound desc = could not find container \"f1826a3f33f87eddcb9b2ec018026ae33445d1878a9fbe3ac94209e6c93851f5\": container with ID starting with f1826a3f33f87eddcb9b2ec018026ae33445d1878a9fbe3ac94209e6c93851f5 not found: ID does not exist"
Sep 17 10:56:08.475811 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.475789    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 10:56:08.476110 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476097    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="config-reloader"
Sep 17 10:56:08.476149 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476112    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="config-reloader"
Sep 17 10:56:08.476149 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476119    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="kube-rbac-proxy-metric"
Sep 17 10:56:08.476149 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476124    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="kube-rbac-proxy-metric"
Sep 17 10:56:08.476149 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476136    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="prom-label-proxy"
Sep 17 10:56:08.476149 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476141    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="prom-label-proxy"
Sep 17 10:56:08.476149 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476147    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97de9f7d-09e9-4412-9c7e-1eb4f2a4d447" containerName="registry"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476153    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="97de9f7d-09e9-4412-9c7e-1eb4f2a4d447" containerName="registry"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476161    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="init-config-reloader"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476166    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="init-config-reloader"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476173    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="alertmanager"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476179    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="alertmanager"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476188    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="kube-rbac-proxy-web"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476193    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="kube-rbac-proxy-web"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476207    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e680891-abe6-43a0-8fe6-22edba5ff5d4" containerName="console"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476212    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e680891-abe6-43a0-8fe6-22edba5ff5d4" containerName="console"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476217    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="kube-rbac-proxy"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476222    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="kube-rbac-proxy"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476273    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="kube-rbac-proxy-web"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476281    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="alertmanager"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476287    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="kube-rbac-proxy-metric"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476293    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="prom-label-proxy"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476299    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="config-reloader"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476304    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="97de9f7d-09e9-4412-9c7e-1eb4f2a4d447" containerName="registry"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476309    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e680891-abe6-43a0-8fe6-22edba5ff5d4" containerName="console"
Sep 17 10:56:08.476343 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.476315    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="266ba384-8031-4300-b02f-9331659c09e2" containerName="kube-rbac-proxy"
Sep 17 10:56:08.481371 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.481351    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.484263 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.484245    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 10:56:08.486382 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.486365    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 10:56:08.486460 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.486428    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 10:56:08.486595 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.486576    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2m2jm\""
Sep 17 10:56:08.486783 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.486768    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 10:56:08.486893 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.486877    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 10:56:08.486963 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.486882    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 10:56:08.486963 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.486948    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 10:56:08.487070 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.487042    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 10:56:08.492403 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.492384    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 10:56:08.499784 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.499763    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 10:56:08.525314 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.525285    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f89b5643-d687-4c4f-af65-5bdb43a25f5b-config-out\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.525460 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.525324    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f89b5643-d687-4c4f-af65-5bdb43a25f5b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.525460 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.525344    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89b5643-d687-4c4f-af65-5bdb43a25f5b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.525460 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.525363    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-config-volume\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.525460 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.525406    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.525460 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.525441    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87lc7\" (UniqueName: \"kubernetes.io/projected/f89b5643-d687-4c4f-af65-5bdb43a25f5b-kube-api-access-87lc7\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.525615 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.525465    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f89b5643-d687-4c4f-af65-5bdb43a25f5b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.525615 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.525499    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f89b5643-d687-4c4f-af65-5bdb43a25f5b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.525615 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.525514    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-web-config\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.525615 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.525563    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.525615 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.525581    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.525615 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.525601    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.525870 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.525621    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.626764 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.626725    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f89b5643-d687-4c4f-af65-5bdb43a25f5b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.626764 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.626762    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89b5643-d687-4c4f-af65-5bdb43a25f5b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.627035 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.626783    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-config-volume\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.627035 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.626831    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.627035 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.626850    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87lc7\" (UniqueName: \"kubernetes.io/projected/f89b5643-d687-4c4f-af65-5bdb43a25f5b-kube-api-access-87lc7\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.627035 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.626867    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f89b5643-d687-4c4f-af65-5bdb43a25f5b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.627243 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.627068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f89b5643-d687-4c4f-af65-5bdb43a25f5b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.627243 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.627110    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-web-config\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.627243 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.627158    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.627243 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.627182    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f89b5643-d687-4c4f-af65-5bdb43a25f5b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.627243 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.627189    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.627243 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.627233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.627527 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.627262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.627527 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.627325    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f89b5643-d687-4c4f-af65-5bdb43a25f5b-config-out\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.628143 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.627768    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f89b5643-d687-4c4f-af65-5bdb43a25f5b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.628769 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.628744    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89b5643-d687-4c4f-af65-5bdb43a25f5b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.629939 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.629917    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f89b5643-d687-4c4f-af65-5bdb43a25f5b-config-out\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.630142 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.630114    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.630249 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.630219    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.630364 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.630262    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-web-config\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.630364 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.630336    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-config-volume\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.630364 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.630346    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.630766 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.630741    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.631457 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.631439    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f89b5643-d687-4c4f-af65-5bdb43a25f5b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.631593 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.631574    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f89b5643-d687-4c4f-af65-5bdb43a25f5b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.635538 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.635518    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87lc7\" (UniqueName: \"kubernetes.io/projected/f89b5643-d687-4c4f-af65-5bdb43a25f5b-kube-api-access-87lc7\") pod \"alertmanager-main-0\" (UID: \"f89b5643-d687-4c4f-af65-5bdb43a25f5b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:08.791013 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:08.790979    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 10:56:09.074218 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:09.074190    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 10:56:09.075772 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:56:09.075748    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89b5643_d687_4c4f_af65_5bdb43a25f5b.slice/crio-e7de5c59e054626b7e2629bfddb34b200c8519607cdb9d60c95b0591ee3a668a WatchSource:0}: Error finding container e7de5c59e054626b7e2629bfddb34b200c8519607cdb9d60c95b0591ee3a668a: Status 404 returned error can't find the container with id e7de5c59e054626b7e2629bfddb34b200c8519607cdb9d60c95b0591ee3a668a
Sep 17 10:56:09.417055 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:09.416963    2562 generic.go:356] "Generic (PLEG): container finished" podID="f89b5643-d687-4c4f-af65-5bdb43a25f5b" containerID="1952730c3932eb6e8e595176f743a37c10c24618821b8930d2c19beb372ea42a" exitCode=0
Sep 17 10:56:09.417198 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:09.417052    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f89b5643-d687-4c4f-af65-5bdb43a25f5b","Type":"ContainerDied","Data":"1952730c3932eb6e8e595176f743a37c10c24618821b8930d2c19beb372ea42a"}
Sep 17 10:56:09.417198 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:09.417131    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f89b5643-d687-4c4f-af65-5bdb43a25f5b","Type":"ContainerStarted","Data":"e7de5c59e054626b7e2629bfddb34b200c8519607cdb9d60c95b0591ee3a668a"}
Sep 17 10:56:09.747872 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:09.747842    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:09.842265 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:09.842233    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:09.946008 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:09.945917    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:10.155242 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:10.155204    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:10.246806 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:10.246699    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:10.425483 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:10.425444    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f89b5643-d687-4c4f-af65-5bdb43a25f5b","Type":"ContainerStarted","Data":"20f7c47bf3fdb396eaec7b887a940301bf8a4c18a8b7609d175c03158f796458"}
Sep 17 10:56:10.425683 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:10.425508    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f89b5643-d687-4c4f-af65-5bdb43a25f5b","Type":"ContainerStarted","Data":"6aff4a2cb50708f24f25cb321c0d60f57162b934241dd25b2d8cb8248aba071e"}
Sep 17 10:56:10.425683 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:10.425536    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f89b5643-d687-4c4f-af65-5bdb43a25f5b","Type":"ContainerStarted","Data":"ffd010528d7499c7f436a16550e9d1da04c5f709e2af6b67019997569dd9d01f"}
Sep 17 10:56:10.425683 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:10.425561    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f89b5643-d687-4c4f-af65-5bdb43a25f5b","Type":"ContainerStarted","Data":"c99174cfff52916cc13d568cc6e15652a7e53400bd6ea34a14aeffe548ef39e3"}
Sep 17 10:56:10.425683 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:10.425583    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f89b5643-d687-4c4f-af65-5bdb43a25f5b","Type":"ContainerStarted","Data":"00c474ecc24fab18c4809ef9298f98a71290b5f9225667cee24f4578c9cae00c"}
Sep 17 10:56:10.425683 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:10.425608    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f89b5643-d687-4c4f-af65-5bdb43a25f5b","Type":"ContainerStarted","Data":"88e8d6f230757deb537338a2375b8da84dc2fc38037164aa44d8965a072e414f"}
Sep 17 10:56:10.463235 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:10.463171    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.446502357 podStartE2EDuration="2.463155071s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.652714ms" isStatefulPod=false podCreationTimestamp="2026-09-17 10:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:56:10.461043179 +0000 UTC m=+164.559088745" watchObservedRunningTime="2026-09-17 10:56:10.463155071 +0000 UTC m=+164.561200677"
Sep 17 10:56:14.461908 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:14.461871    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:14.630783 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:14.630748    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:21.081582 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:21.081496    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:21.181564 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:21.181528    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:21.283492 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:21.283463    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:22.585248 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:22.585216    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:22.733591 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:22.733557    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:45.561545 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:45.561512    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:45.669404 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:45.669373    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:46.971126 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:46.971092    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:47.111366 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:47.111335    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:53.631205 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:53.631169    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:56.163215 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:56.163172    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:56:56.241998 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:56:56.241960    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:10.407524 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:10.407493    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:10.751345 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:10.751268    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:14.629598 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:14.629557    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:14.810434 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:14.810399    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:15.588521 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:15.588491    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:15.775192 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:15.775156    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:16.809396 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:16.809365    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:16.943741 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:16.943710    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:19.188242 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:19.188203    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:19.285278 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:19.285244    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9fnhj"]
Sep 17 10:57:19.288396 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:19.288378    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9fnhj"
Sep 17 10:57:19.291397 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:19.291374    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 10:57:19.291510 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:19.291490    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-xrkmx\""
Sep 17 10:57:19.292551 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:19.292536    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 10:57:19.310123 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:19.310092    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9fnhj"]
Sep 17 10:57:19.397143 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:19.397112    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7481f9fd-b664-4aee-95c5-0d805dcd1e14-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9fnhj\" (UID: \"7481f9fd-b664-4aee-95c5-0d805dcd1e14\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9fnhj"
Sep 17 10:57:19.397296 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:19.397160    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhb9r\" (UniqueName: \"kubernetes.io/projected/7481f9fd-b664-4aee-95c5-0d805dcd1e14-kube-api-access-xhb9r\") pod \"cert-manager-cainjector-559cdd8c87-9fnhj\" (UID: \"7481f9fd-b664-4aee-95c5-0d805dcd1e14\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9fnhj"
Sep 17 10:57:19.440029 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:19.439937    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:19.498396 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:19.498353    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7481f9fd-b664-4aee-95c5-0d805dcd1e14-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9fnhj\" (UID: \"7481f9fd-b664-4aee-95c5-0d805dcd1e14\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9fnhj"
Sep 17 10:57:19.498548 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:19.498420    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhb9r\" (UniqueName: \"kubernetes.io/projected/7481f9fd-b664-4aee-95c5-0d805dcd1e14-kube-api-access-xhb9r\") pod \"cert-manager-cainjector-559cdd8c87-9fnhj\" (UID: \"7481f9fd-b664-4aee-95c5-0d805dcd1e14\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9fnhj"
Sep 17 10:57:19.507695 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:19.507663    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhb9r\" (UniqueName: \"kubernetes.io/projected/7481f9fd-b664-4aee-95c5-0d805dcd1e14-kube-api-access-xhb9r\") pod \"cert-manager-cainjector-559cdd8c87-9fnhj\" (UID: \"7481f9fd-b664-4aee-95c5-0d805dcd1e14\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9fnhj"
Sep 17 10:57:19.508673 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:19.508651    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7481f9fd-b664-4aee-95c5-0d805dcd1e14-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9fnhj\" (UID: \"7481f9fd-b664-4aee-95c5-0d805dcd1e14\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9fnhj"
Sep 17 10:57:19.616152 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:19.616115    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9fnhj"
Sep 17 10:57:19.894018 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:19.893991    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9fnhj"]
Sep 17 10:57:19.896364 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:57:19.896336    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7481f9fd_b664_4aee_95c5_0d805dcd1e14.slice/crio-7c0573f37825438bc2629bce05ed981a99f0aa29dda6420ab2d7ce1a8cd91c63 WatchSource:0}: Error finding container 7c0573f37825438bc2629bce05ed981a99f0aa29dda6420ab2d7ce1a8cd91c63: Status 404 returned error can't find the container with id 7c0573f37825438bc2629bce05ed981a99f0aa29dda6420ab2d7ce1a8cd91c63
Sep 17 10:57:20.708993 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:20.708949    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9fnhj" event={"ID":"7481f9fd-b664-4aee-95c5-0d805dcd1e14","Type":"ContainerStarted","Data":"7c0573f37825438bc2629bce05ed981a99f0aa29dda6420ab2d7ce1a8cd91c63"}
Sep 17 10:57:21.209341 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:21.209261    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:21.282439 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:21.282357    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:23.723861 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:23.723810    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9fnhj" event={"ID":"7481f9fd-b664-4aee-95c5-0d805dcd1e14","Type":"ContainerStarted","Data":"3a419700df7e9674eff8899b529874d998d5d6aa4731fc7c5338cf377335561c"}
Sep 17 10:57:23.752239 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:23.752182    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9fnhj" podStartSLOduration=1.647099555 podStartE2EDuration="4.752164403s" totalImagesPullingTime="3.105064848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:57:23.749425473 +0000 UTC m=+237.847471039" watchObservedRunningTime="2026-09-17 10:57:23.752164403 +0000 UTC m=+237.850209968"
Sep 17 10:57:23.811763 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:23.811736    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:23.908097 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:23.908068    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:26.783344 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:26.783315    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:27.967612 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:27.967578    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:28.146962 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:28.146919    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:29.385850 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:29.385799    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fkhcc"]
Sep 17 10:57:29.389075 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:29.389056    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fkhcc"
Sep 17 10:57:29.392223 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:29.392200    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-pjx54\""
Sep 17 10:57:29.392353 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:29.392243    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 10:57:29.393355 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:29.393335    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 10:57:29.409937 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:29.409912    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fkhcc"]
Sep 17 10:57:29.484258 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:29.484224    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aad44ce7-e30e-48e2-aa16-6e93398ea89f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fkhcc\" (UID: \"aad44ce7-e30e-48e2-aa16-6e93398ea89f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fkhcc"
Sep 17 10:57:29.484445 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:29.484277    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tmml\" (UniqueName: \"kubernetes.io/projected/aad44ce7-e30e-48e2-aa16-6e93398ea89f-kube-api-access-4tmml\") pod \"openshift-lws-operator-fd8ccff4c-fkhcc\" (UID: \"aad44ce7-e30e-48e2-aa16-6e93398ea89f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fkhcc"
Sep 17 10:57:29.585588 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:29.585548    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aad44ce7-e30e-48e2-aa16-6e93398ea89f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fkhcc\" (UID: \"aad44ce7-e30e-48e2-aa16-6e93398ea89f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fkhcc"
Sep 17 10:57:29.585787 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:29.585621    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tmml\" (UniqueName: \"kubernetes.io/projected/aad44ce7-e30e-48e2-aa16-6e93398ea89f-kube-api-access-4tmml\") pod \"openshift-lws-operator-fd8ccff4c-fkhcc\" (UID: \"aad44ce7-e30e-48e2-aa16-6e93398ea89f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fkhcc"
Sep 17 10:57:29.585971 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:29.585948    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aad44ce7-e30e-48e2-aa16-6e93398ea89f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fkhcc\" (UID: \"aad44ce7-e30e-48e2-aa16-6e93398ea89f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fkhcc"
Sep 17 10:57:29.600358 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:29.600329    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tmml\" (UniqueName: \"kubernetes.io/projected/aad44ce7-e30e-48e2-aa16-6e93398ea89f-kube-api-access-4tmml\") pod \"openshift-lws-operator-fd8ccff4c-fkhcc\" (UID: \"aad44ce7-e30e-48e2-aa16-6e93398ea89f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fkhcc"
Sep 17 10:57:29.697371 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:29.697289    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fkhcc"
Sep 17 10:57:29.976110 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:29.976083    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fkhcc"]
Sep 17 10:57:29.978333 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:57:29.978301    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaad44ce7_e30e_48e2_aa16_6e93398ea89f.slice/crio-6ab12ab99dfd5b5e011639793847a84e3c3c556e14401ca179b57f5ccfdbac8c WatchSource:0}: Error finding container 6ab12ab99dfd5b5e011639793847a84e3c3c556e14401ca179b57f5ccfdbac8c: Status 404 returned error can't find the container with id 6ab12ab99dfd5b5e011639793847a84e3c3c556e14401ca179b57f5ccfdbac8c
Sep 17 10:57:30.755354 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:30.755320    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fkhcc" event={"ID":"aad44ce7-e30e-48e2-aa16-6e93398ea89f","Type":"ContainerStarted","Data":"6ab12ab99dfd5b5e011639793847a84e3c3c556e14401ca179b57f5ccfdbac8c"}
Sep 17 10:57:33.771229 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:33.771193    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fkhcc" event={"ID":"aad44ce7-e30e-48e2-aa16-6e93398ea89f","Type":"ContainerStarted","Data":"d4539cbea8ccac71aadd8c371fa10a2c46d06ce10c1bc9b3edf024d11bfbd444"}
Sep 17 10:57:33.794071 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:33.794019    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fkhcc" podStartSLOduration=1.940473788 podStartE2EDuration="4.794004255s" totalImagesPullingTime="2.853530467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:57:33.792247385 +0000 UTC m=+247.890292979" watchObservedRunningTime="2026-09-17 10:57:33.794004255 +0000 UTC m=+247.892049822"
Sep 17 10:57:34.624189 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:34.624151    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:34.735709 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:34.735669    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:36.416694 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:36.416656    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:48.272669 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:48.272630    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:48.379213 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:48.379176    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:48.709753 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:48.709716    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:49.020481 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.020375    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:49.127159 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.127122    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:49.203125 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.203085    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"]
Sep 17 10:57:49.206488 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.206465    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:57:49.212168 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.212134    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ff97h\""
Sep 17 10:57:49.213205 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.213081    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 17 10:57:49.213558 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.213541    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 17 10:57:49.215225 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.215211    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 17 10:57:49.216937 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.216922    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 17 10:57:49.245670 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.244069    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"]
Sep 17 10:57:49.357702 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.357666    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e-webhook-cert\") pod \"opendatahub-operator-controller-manager-c54c4bd97-sjrlr\" (UID: \"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e\") " pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:57:49.357702 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.357715    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdbk6\" (UniqueName: \"kubernetes.io/projected/9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e-kube-api-access-hdbk6\") pod \"opendatahub-operator-controller-manager-c54c4bd97-sjrlr\" (UID: \"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e\") " pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:57:49.358175 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.357750    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c54c4bd97-sjrlr\" (UID: \"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e\") " pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:57:49.358175 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.357857    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e-manifests\") pod \"opendatahub-operator-controller-manager-c54c4bd97-sjrlr\" (UID: \"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e\") " pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:57:49.358175 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.357895    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e-tmp\") pod \"opendatahub-operator-controller-manager-c54c4bd97-sjrlr\" (UID: \"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e\") " pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:57:49.459100 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.459059    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e-webhook-cert\") pod \"opendatahub-operator-controller-manager-c54c4bd97-sjrlr\" (UID: \"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e\") " pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:57:49.459100 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.459108    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdbk6\" (UniqueName: \"kubernetes.io/projected/9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e-kube-api-access-hdbk6\") pod \"opendatahub-operator-controller-manager-c54c4bd97-sjrlr\" (UID: \"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e\") " pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:57:49.459319 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.459146    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c54c4bd97-sjrlr\" (UID: \"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e\") " pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:57:49.459319 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.459194    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e-manifests\") pod \"opendatahub-operator-controller-manager-c54c4bd97-sjrlr\" (UID: \"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e\") " pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:57:49.459319 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.459220    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e-tmp\") pod \"opendatahub-operator-controller-manager-c54c4bd97-sjrlr\" (UID: \"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e\") " pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:57:49.459655 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.459621    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e-manifests\") pod \"opendatahub-operator-controller-manager-c54c4bd97-sjrlr\" (UID: \"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e\") " pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:57:49.461486 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.461467    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e-tmp\") pod \"opendatahub-operator-controller-manager-c54c4bd97-sjrlr\" (UID: \"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e\") " pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:57:49.461617 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.461598    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e-webhook-cert\") pod \"opendatahub-operator-controller-manager-c54c4bd97-sjrlr\" (UID: \"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e\") " pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:57:49.461849 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.461799    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c54c4bd97-sjrlr\" (UID: \"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e\") " pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:57:49.468135 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.468111    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdbk6\" (UniqueName: \"kubernetes.io/projected/9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e-kube-api-access-hdbk6\") pod \"opendatahub-operator-controller-manager-c54c4bd97-sjrlr\" (UID: \"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e\") " pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:57:49.480526 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.480498    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:49.516606 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.516567    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:57:49.580590 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.580380    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:49.684100 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.684018    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:49.781453 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.781409    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:49.807190 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.807167    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"]
Sep 17 10:57:49.809234 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:57:49.809201    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc68714_6c9b_4732_8c7c_9fa5fb1ae69e.slice/crio-86ef03045a8845218d6d6a93104b6a1cb21135012df3c71f3da822d164b9add9 WatchSource:0}: Error finding container 86ef03045a8845218d6d6a93104b6a1cb21135012df3c71f3da822d164b9add9: Status 404 returned error can't find the container with id 86ef03045a8845218d6d6a93104b6a1cb21135012df3c71f3da822d164b9add9
Sep 17 10:57:49.843498 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.843460    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr" event={"ID":"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e","Type":"ContainerStarted","Data":"86ef03045a8845218d6d6a93104b6a1cb21135012df3c71f3da822d164b9add9"}
Sep 17 10:57:49.885077 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.885035    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:49.992961 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:49.992873    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:50.090643 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:50.090604    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:57:52.859805 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:52.859768    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr" event={"ID":"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e","Type":"ContainerStarted","Data":"4d56d3ce84e6ad46ce1ce792396c7da5c165bf789bb5f47efa85e9c320ec39d3"}
Sep 17 10:57:53.865621 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:53.865593    2562 generic.go:356] "Generic (PLEG): container finished" podID="9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e" containerID="4d56d3ce84e6ad46ce1ce792396c7da5c165bf789bb5f47efa85e9c320ec39d3" exitCode=0
Sep 17 10:57:53.866365 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:53.865636    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr" event={"ID":"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e","Type":"ContainerDied","Data":"4d56d3ce84e6ad46ce1ce792396c7da5c165bf789bb5f47efa85e9c320ec39d3"}
Sep 17 10:57:54.873528 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:54.873485    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr" event={"ID":"9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e","Type":"ContainerStarted","Data":"b1f9d77dd2bd5a99f3426da9a789d7775aad90fc44cc9a408c4a51eb1d265ec0"}
Sep 17 10:57:54.874019 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:54.873616    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:57:54.901280 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:57:54.901229    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr" podStartSLOduration=1.576263087 podStartE2EDuration="5.901214428s" totalImagesPullingTime="3.294862773s" totalInitContainerRuntime="1.030088568s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:57:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:57:54.899432706 +0000 UTC m=+268.997478283" watchObservedRunningTime="2026-09-17 10:57:54.901214428 +0000 UTC m=+268.999259994"
Sep 17 10:58:04.143251 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:04.143217    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:04.272442 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:04.272409    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:05.880512 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:05.880482    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c54c4bd97-sjrlr"
Sep 17 10:58:06.171285 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.171205    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"]
Sep 17 10:58:06.182226 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.182199    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"
Sep 17 10:58:06.191204 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.191180    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 10:58:06.191481 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.191453    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-djmd9\""
Sep 17 10:58:06.191570 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.191507    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 10:58:06.191914 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.191902    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 10:58:06.194276 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.194259    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/908cfe1e-afa6-44f7-bae2-c8fe5f8ba786-manager-config\") pod \"lws-controller-manager-7bb4f45b46-2gcl9\" (UID: \"908cfe1e-afa6-44f7-bae2-c8fe5f8ba786\") " pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"
Sep 17 10:58:06.194354 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.194298    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mj9f\" (UniqueName: \"kubernetes.io/projected/908cfe1e-afa6-44f7-bae2-c8fe5f8ba786-kube-api-access-5mj9f\") pod \"lws-controller-manager-7bb4f45b46-2gcl9\" (UID: \"908cfe1e-afa6-44f7-bae2-c8fe5f8ba786\") " pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"
Sep 17 10:58:06.194411 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.194352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/908cfe1e-afa6-44f7-bae2-c8fe5f8ba786-metrics-cert\") pod \"lws-controller-manager-7bb4f45b46-2gcl9\" (UID: \"908cfe1e-afa6-44f7-bae2-c8fe5f8ba786\") " pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"
Sep 17 10:58:06.194411 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.194396    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/908cfe1e-afa6-44f7-bae2-c8fe5f8ba786-cert\") pod \"lws-controller-manager-7bb4f45b46-2gcl9\" (UID: \"908cfe1e-afa6-44f7-bae2-c8fe5f8ba786\") " pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"
Sep 17 10:58:06.201013 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.200988    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"]
Sep 17 10:58:06.295090 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.295057    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/908cfe1e-afa6-44f7-bae2-c8fe5f8ba786-manager-config\") pod \"lws-controller-manager-7bb4f45b46-2gcl9\" (UID: \"908cfe1e-afa6-44f7-bae2-c8fe5f8ba786\") " pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"
Sep 17 10:58:06.295257 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.295114    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mj9f\" (UniqueName: \"kubernetes.io/projected/908cfe1e-afa6-44f7-bae2-c8fe5f8ba786-kube-api-access-5mj9f\") pod \"lws-controller-manager-7bb4f45b46-2gcl9\" (UID: \"908cfe1e-afa6-44f7-bae2-c8fe5f8ba786\") " pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"
Sep 17 10:58:06.295257 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.295140    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/908cfe1e-afa6-44f7-bae2-c8fe5f8ba786-metrics-cert\") pod \"lws-controller-manager-7bb4f45b46-2gcl9\" (UID: \"908cfe1e-afa6-44f7-bae2-c8fe5f8ba786\") " pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"
Sep 17 10:58:06.295257 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.295168    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/908cfe1e-afa6-44f7-bae2-c8fe5f8ba786-cert\") pod \"lws-controller-manager-7bb4f45b46-2gcl9\" (UID: \"908cfe1e-afa6-44f7-bae2-c8fe5f8ba786\") " pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"
Sep 17 10:58:06.295800 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.295775    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/908cfe1e-afa6-44f7-bae2-c8fe5f8ba786-manager-config\") pod \"lws-controller-manager-7bb4f45b46-2gcl9\" (UID: \"908cfe1e-afa6-44f7-bae2-c8fe5f8ba786\") " pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"
Sep 17 10:58:06.297611 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.297594    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/908cfe1e-afa6-44f7-bae2-c8fe5f8ba786-metrics-cert\") pod \"lws-controller-manager-7bb4f45b46-2gcl9\" (UID: \"908cfe1e-afa6-44f7-bae2-c8fe5f8ba786\") " pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"
Sep 17 10:58:06.297666 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.297618    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/908cfe1e-afa6-44f7-bae2-c8fe5f8ba786-cert\") pod \"lws-controller-manager-7bb4f45b46-2gcl9\" (UID: \"908cfe1e-afa6-44f7-bae2-c8fe5f8ba786\") " pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"
Sep 17 10:58:06.304054 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.304034    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mj9f\" (UniqueName: \"kubernetes.io/projected/908cfe1e-afa6-44f7-bae2-c8fe5f8ba786-kube-api-access-5mj9f\") pod \"lws-controller-manager-7bb4f45b46-2gcl9\" (UID: \"908cfe1e-afa6-44f7-bae2-c8fe5f8ba786\") " pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"
Sep 17 10:58:06.491000 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.490910    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"
Sep 17 10:58:06.768109 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.768055    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"]
Sep 17 10:58:06.770238 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:58:06.770213    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908cfe1e_afa6_44f7_bae2_c8fe5f8ba786.slice/crio-f38ea3bdcc49330380751e65f10e7e9fc42bb3fb356bc14661c3a8169c277f58 WatchSource:0}: Error finding container f38ea3bdcc49330380751e65f10e7e9fc42bb3fb356bc14661c3a8169c277f58: Status 404 returned error can't find the container with id f38ea3bdcc49330380751e65f10e7e9fc42bb3fb356bc14661c3a8169c277f58
Sep 17 10:58:06.931108 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:06.931066    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9" event={"ID":"908cfe1e-afa6-44f7-bae2-c8fe5f8ba786","Type":"ContainerStarted","Data":"f38ea3bdcc49330380751e65f10e7e9fc42bb3fb356bc14661c3a8169c277f58"}
Sep 17 10:58:09.130361 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:09.130327    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:09.286372 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:09.286335    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:09.946550 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:09.946514    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9" event={"ID":"908cfe1e-afa6-44f7-bae2-c8fe5f8ba786","Type":"ContainerStarted","Data":"3287a8162ae38fae450e64ed6dd7bfccdf7cc4c434e69e14b2586263cf4bf2cd"}
Sep 17 10:58:09.946729 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:09.946575    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"
Sep 17 10:58:09.990501 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:09.990445    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9" podStartSLOduration=1.638422264 podStartE2EDuration="3.990429697s" totalImagesPullingTime="2.352007433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:58:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:58:09.988124035 +0000 UTC m=+284.086169602" watchObservedRunningTime="2026-09-17 10:58:09.990429697 +0000 UTC m=+284.088475262"
Sep 17 10:58:12.210236 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:12.210202    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:12.245380 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:12.245344    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8665f874d8-p5sj7"]
Sep 17 10:58:12.248879 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:12.248849    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8665f874d8-p5sj7"
Sep 17 10:58:12.254346 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:12.254325    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 17 10:58:12.254484 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:12.254414    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7k5w5\""
Sep 17 10:58:12.254484 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:12.254416    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 17 10:58:12.275518 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:12.275482    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8665f874d8-p5sj7"]
Sep 17 10:58:12.340275 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:12.340239    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ad2f5ff-7402-4b66-aae0-606b585eb376-tmp\") pod \"kube-auth-proxy-8665f874d8-p5sj7\" (UID: \"1ad2f5ff-7402-4b66-aae0-606b585eb376\") " pod="openshift-ingress/kube-auth-proxy-8665f874d8-p5sj7"
Sep 17 10:58:12.340455 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:12.340286    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ad2f5ff-7402-4b66-aae0-606b585eb376-tls-certs\") pod \"kube-auth-proxy-8665f874d8-p5sj7\" (UID: \"1ad2f5ff-7402-4b66-aae0-606b585eb376\") " pod="openshift-ingress/kube-auth-proxy-8665f874d8-p5sj7"
Sep 17 10:58:12.340455 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:12.340343    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svpkm\" (UniqueName: \"kubernetes.io/projected/1ad2f5ff-7402-4b66-aae0-606b585eb376-kube-api-access-svpkm\") pod \"kube-auth-proxy-8665f874d8-p5sj7\" (UID: \"1ad2f5ff-7402-4b66-aae0-606b585eb376\") " pod="openshift-ingress/kube-auth-proxy-8665f874d8-p5sj7"
Sep 17 10:58:12.441631 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:12.441596    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svpkm\" (UniqueName: \"kubernetes.io/projected/1ad2f5ff-7402-4b66-aae0-606b585eb376-kube-api-access-svpkm\") pod \"kube-auth-proxy-8665f874d8-p5sj7\" (UID: \"1ad2f5ff-7402-4b66-aae0-606b585eb376\") " pod="openshift-ingress/kube-auth-proxy-8665f874d8-p5sj7"
Sep 17 10:58:12.441631 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:12.441632    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ad2f5ff-7402-4b66-aae0-606b585eb376-tmp\") pod \"kube-auth-proxy-8665f874d8-p5sj7\" (UID: \"1ad2f5ff-7402-4b66-aae0-606b585eb376\") " pod="openshift-ingress/kube-auth-proxy-8665f874d8-p5sj7"
Sep 17 10:58:12.441904 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:12.441666    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ad2f5ff-7402-4b66-aae0-606b585eb376-tls-certs\") pod \"kube-auth-proxy-8665f874d8-p5sj7\" (UID: \"1ad2f5ff-7402-4b66-aae0-606b585eb376\") " pod="openshift-ingress/kube-auth-proxy-8665f874d8-p5sj7"
Sep 17 10:58:12.441904 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:58:12.441793    2562 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 17 10:58:12.441904 ip-10-0-139-204 kubenswrapper[2562]: E0917 10:58:12.441861    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ad2f5ff-7402-4b66-aae0-606b585eb376-tls-certs podName:1ad2f5ff-7402-4b66-aae0-606b585eb376 nodeName:}" failed. No retries permitted until 2026-09-17 10:58:12.941846276 +0000 UTC m=+287.039891821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1ad2f5ff-7402-4b66-aae0-606b585eb376-tls-certs") pod "kube-auth-proxy-8665f874d8-p5sj7" (UID: "1ad2f5ff-7402-4b66-aae0-606b585eb376") : secret "kube-auth-proxy-tls" not found
Sep 17 10:58:12.443952 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:12.443933    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ad2f5ff-7402-4b66-aae0-606b585eb376-tmp\") pod \"kube-auth-proxy-8665f874d8-p5sj7\" (UID: \"1ad2f5ff-7402-4b66-aae0-606b585eb376\") " pod="openshift-ingress/kube-auth-proxy-8665f874d8-p5sj7"
Sep 17 10:58:12.451288 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:12.451261    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svpkm\" (UniqueName: \"kubernetes.io/projected/1ad2f5ff-7402-4b66-aae0-606b585eb376-kube-api-access-svpkm\") pod \"kube-auth-proxy-8665f874d8-p5sj7\" (UID: \"1ad2f5ff-7402-4b66-aae0-606b585eb376\") " pod="openshift-ingress/kube-auth-proxy-8665f874d8-p5sj7"
Sep 17 10:58:12.517534 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:12.517454    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:12.944664 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:12.944619    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ad2f5ff-7402-4b66-aae0-606b585eb376-tls-certs\") pod \"kube-auth-proxy-8665f874d8-p5sj7\" (UID: \"1ad2f5ff-7402-4b66-aae0-606b585eb376\") " pod="openshift-ingress/kube-auth-proxy-8665f874d8-p5sj7"
Sep 17 10:58:12.947010 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:12.946981    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ad2f5ff-7402-4b66-aae0-606b585eb376-tls-certs\") pod \"kube-auth-proxy-8665f874d8-p5sj7\" (UID: \"1ad2f5ff-7402-4b66-aae0-606b585eb376\") " pod="openshift-ingress/kube-auth-proxy-8665f874d8-p5sj7"
Sep 17 10:58:13.161218 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:13.161182    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8665f874d8-p5sj7"
Sep 17 10:58:13.448314 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:13.448286    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8665f874d8-p5sj7"]
Sep 17 10:58:13.450809 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:58:13.450738    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad2f5ff_7402_4b66_aae0_606b585eb376.slice/crio-7e8f4f00cca36e6ba4bf85b054931b1898fad5dc787e975a8517b6b28a3c2803 WatchSource:0}: Error finding container 7e8f4f00cca36e6ba4bf85b054931b1898fad5dc787e975a8517b6b28a3c2803: Status 404 returned error can't find the container with id 7e8f4f00cca36e6ba4bf85b054931b1898fad5dc787e975a8517b6b28a3c2803
Sep 17 10:58:13.797040 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:13.797009    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:13.870626 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:13.870586    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:13.965393 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:13.965360    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8665f874d8-p5sj7" event={"ID":"1ad2f5ff-7402-4b66-aae0-606b585eb376","Type":"ContainerStarted","Data":"7e8f4f00cca36e6ba4bf85b054931b1898fad5dc787e975a8517b6b28a3c2803"}
Sep 17 10:58:15.326955 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:15.326924    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:16.983080 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:16.983044    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8665f874d8-p5sj7" event={"ID":"1ad2f5ff-7402-4b66-aae0-606b585eb376","Type":"ContainerStarted","Data":"4e2bc7bd60e88662f723889b50fb287e03f56aeeacad4e31670ef6226396d419"}
Sep 17 10:58:17.007467 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:17.007413    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8665f874d8-p5sj7" podStartSLOduration=1.941403372 podStartE2EDuration="5.00739795s" totalImagesPullingTime="3.065994578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:58:17.00615442 +0000 UTC m=+291.104199988" watchObservedRunningTime="2026-09-17 10:58:17.00739795 +0000 UTC m=+291.105443515"
Sep 17 10:58:18.343445 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:18.343408    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:18.460447 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:18.460417    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:19.411890 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:19.411853    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:19.628191 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:19.628159    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:19.732677 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:19.732593    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:20.498789 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:20.498758    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:20.728078 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:20.728050    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:20.836268 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:20.836232    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:20.934873 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:20.934840    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:20.953573 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:20.953535    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7bb4f45b46-2gcl9"
Sep 17 10:58:21.189830 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:21.189715    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:21.458590 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:21.458505    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:26.334871 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:26.334835    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 10:58:26.342017 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:26.341987    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 10:58:26.352590 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:26.352559    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 10:58:30.688991 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:30.688953    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:32.318019 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:32.317983    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:32.453360 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:32.453331    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:58:32.638763 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:58:32.638671    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:21.743483 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:21.743410    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:21.842507 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:21.842470    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:37.987504 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:37.987460    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:38.785226 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:38.785183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:41.780600 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:41.780563    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:42.231885 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:42.231784    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:42.613781 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:42.613738    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:43.022618 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:43.022506    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:43.993627 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:43.993600    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:44.245398 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:44.245287    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:46.492070 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:46.491990    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:48.500701 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:48.500630    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:48.663944 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:48.663906    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:49.903862 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:49.903801    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:52.164846 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:52.164785    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:52.351626 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:52.351596    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:52.541601 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:52.541568    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:52.638213 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:52.638186    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:54.565657 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.565624    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54cc58f9b9-28cjv"]
Sep 17 10:59:54.569386 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.569363    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.572469 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.572444    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 10:59:54.573707 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.573688    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 10:59:54.573840 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.573722    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 10:59:54.574027 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.574010    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qxbsw\""
Sep 17 10:59:54.574542 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.574521    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 10:59:54.574656 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.574635    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 10:59:54.580259 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.580238    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 10:59:54.584065 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.584043    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54cc58f9b9-28cjv"]
Sep 17 10:59:54.658198 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.658167    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:54.723417 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.723375    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f5fe193-073f-4040-9149-cccb6f46a18e-trusted-ca-bundle\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.723629 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.723425    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f5fe193-073f-4040-9149-cccb6f46a18e-console-serving-cert\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.723629 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.723459    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f5fe193-073f-4040-9149-cccb6f46a18e-console-config\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.723629 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.723499    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5fe193-073f-4040-9149-cccb6f46a18e-service-ca\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.723629 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.723524    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f5fe193-073f-4040-9149-cccb6f46a18e-console-oauth-config\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.723629 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.723573    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2h8t\" (UniqueName: \"kubernetes.io/projected/8f5fe193-073f-4040-9149-cccb6f46a18e-kube-api-access-b2h8t\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.723629 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.723593    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f5fe193-073f-4040-9149-cccb6f46a18e-oauth-serving-cert\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.812470 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.812442    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:54.824864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.824780    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f5fe193-073f-4040-9149-cccb6f46a18e-trusted-ca-bundle\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.824864 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.824851    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f5fe193-073f-4040-9149-cccb6f46a18e-console-serving-cert\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.825030 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.824885    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f5fe193-073f-4040-9149-cccb6f46a18e-console-config\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.825030 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.824945    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5fe193-073f-4040-9149-cccb6f46a18e-service-ca\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.825030 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.824981    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f5fe193-073f-4040-9149-cccb6f46a18e-console-oauth-config\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.825030 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.825014    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2h8t\" (UniqueName: \"kubernetes.io/projected/8f5fe193-073f-4040-9149-cccb6f46a18e-kube-api-access-b2h8t\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.825237 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.825039    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f5fe193-073f-4040-9149-cccb6f46a18e-oauth-serving-cert\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.825758 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.825719    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f5fe193-073f-4040-9149-cccb6f46a18e-trusted-ca-bundle\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.825906 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.825834    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f5fe193-073f-4040-9149-cccb6f46a18e-console-config\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.825906 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.825860    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5fe193-073f-4040-9149-cccb6f46a18e-service-ca\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.825998 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.825913    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f5fe193-073f-4040-9149-cccb6f46a18e-oauth-serving-cert\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.827643 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.827621    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f5fe193-073f-4040-9149-cccb6f46a18e-console-serving-cert\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.828230 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.828215    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f5fe193-073f-4040-9149-cccb6f46a18e-console-oauth-config\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.839339 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.839302    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2h8t\" (UniqueName: \"kubernetes.io/projected/8f5fe193-073f-4040-9149-cccb6f46a18e-kube-api-access-b2h8t\") pod \"console-54cc58f9b9-28cjv\" (UID: \"8f5fe193-073f-4040-9149-cccb6f46a18e\") " pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.880184 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.880151    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 10:59:54.889140 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.889116    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-n598x"]
Sep 17 10:59:54.892496 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.892477    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n598x"
Sep 17 10:59:54.896431 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.896404    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 17 10:59:54.896873 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.896856    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-262zf\""
Sep 17 10:59:54.896970 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.896929    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 10:59:54.897160 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.897132    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 17 10:59:54.897595 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.897581    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 10:59:54.911875 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.911847    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-n598x"]
Sep 17 10:59:54.917995 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.917966    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:54.925539 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.925505    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3a3cc953-5ff2-42ca-a53f-ae05c57581e3-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-n598x\" (UID: \"3a3cc953-5ff2-42ca-a53f-ae05c57581e3\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n598x"
Sep 17 10:59:54.925653 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.925621    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v2d7\" (UniqueName: \"kubernetes.io/projected/3a3cc953-5ff2-42ca-a53f-ae05c57581e3-kube-api-access-6v2d7\") pod \"kuadrant-console-plugin-fbdd4f445-n598x\" (UID: \"3a3cc953-5ff2-42ca-a53f-ae05c57581e3\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n598x"
Sep 17 10:59:54.925736 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:54.925709    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a3cc953-5ff2-42ca-a53f-ae05c57581e3-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-n598x\" (UID: \"3a3cc953-5ff2-42ca-a53f-ae05c57581e3\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n598x"
Sep 17 10:59:55.026756 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:55.026727    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3a3cc953-5ff2-42ca-a53f-ae05c57581e3-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-n598x\" (UID: \"3a3cc953-5ff2-42ca-a53f-ae05c57581e3\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n598x"
Sep 17 10:59:55.026947 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:55.026793    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2d7\" (UniqueName: \"kubernetes.io/projected/3a3cc953-5ff2-42ca-a53f-ae05c57581e3-kube-api-access-6v2d7\") pod \"kuadrant-console-plugin-fbdd4f445-n598x\" (UID: \"3a3cc953-5ff2-42ca-a53f-ae05c57581e3\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n598x"
Sep 17 10:59:55.027029 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:55.027004    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a3cc953-5ff2-42ca-a53f-ae05c57581e3-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-n598x\" (UID: \"3a3cc953-5ff2-42ca-a53f-ae05c57581e3\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n598x"
Sep 17 10:59:55.027423 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:55.027397    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3a3cc953-5ff2-42ca-a53f-ae05c57581e3-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-n598x\" (UID: \"3a3cc953-5ff2-42ca-a53f-ae05c57581e3\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n598x"
Sep 17 10:59:55.029335 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:55.029315    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a3cc953-5ff2-42ca-a53f-ae05c57581e3-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-n598x\" (UID: \"3a3cc953-5ff2-42ca-a53f-ae05c57581e3\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n598x"
Sep 17 10:59:55.036572 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:55.036540    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v2d7\" (UniqueName: \"kubernetes.io/projected/3a3cc953-5ff2-42ca-a53f-ae05c57581e3-kube-api-access-6v2d7\") pod \"kuadrant-console-plugin-fbdd4f445-n598x\" (UID: \"3a3cc953-5ff2-42ca-a53f-ae05c57581e3\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n598x"
Sep 17 10:59:55.055408 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:55.055344    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:55.169176 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:55.169095    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54cc58f9b9-28cjv"]
Sep 17 10:59:55.172284 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:59:55.172257    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5fe193_073f_4040_9149_cccb6f46a18e.slice/crio-9885e9ff1da32f2f54701f7f187bead0103e2d8b0a3b304ae1d8ae5804e43f2b WatchSource:0}: Error finding container 9885e9ff1da32f2f54701f7f187bead0103e2d8b0a3b304ae1d8ae5804e43f2b: Status 404 returned error can't find the container with id 9885e9ff1da32f2f54701f7f187bead0103e2d8b0a3b304ae1d8ae5804e43f2b
Sep 17 10:59:55.219117 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:55.219095    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n598x"
Sep 17 10:59:55.438318 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:55.438238    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cc58f9b9-28cjv" event={"ID":"8f5fe193-073f-4040-9149-cccb6f46a18e","Type":"ContainerStarted","Data":"df7aef9524f9cf67df4a6930cb4e44b47b0aba1e1b973be48bd370e3c1cbd114"}
Sep 17 10:59:55.438318 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:55.438284    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cc58f9b9-28cjv" event={"ID":"8f5fe193-073f-4040-9149-cccb6f46a18e","Type":"ContainerStarted","Data":"9885e9ff1da32f2f54701f7f187bead0103e2d8b0a3b304ae1d8ae5804e43f2b"}
Sep 17 10:59:55.473234 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:55.473180    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54cc58f9b9-28cjv" podStartSLOduration=1.473164533 podStartE2EDuration="1.473164533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 10:59:55.470690175 +0000 UTC m=+389.568735742" watchObservedRunningTime="2026-09-17 10:59:55.473164533 +0000 UTC m=+389.571210099"
Sep 17 10:59:55.496334 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:55.496312    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-n598x"]
Sep 17 10:59:55.498452 ip-10-0-139-204 kubenswrapper[2562]: W0917 10:59:55.498426    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a3cc953_5ff2_42ca_a53f_ae05c57581e3.slice/crio-9efe3ba6d1665a2d73ef2683bf59ad2f1634f8dcad9c8c8d532515c025321106 WatchSource:0}: Error finding container 9efe3ba6d1665a2d73ef2683bf59ad2f1634f8dcad9c8c8d532515c025321106: Status 404 returned error can't find the container with id 9efe3ba6d1665a2d73ef2683bf59ad2f1634f8dcad9c8c8d532515c025321106
Sep 17 10:59:55.499862 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:55.499844    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 10:59:56.457467 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:56.457424    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n598x" event={"ID":"3a3cc953-5ff2-42ca-a53f-ae05c57581e3","Type":"ContainerStarted","Data":"9efe3ba6d1665a2d73ef2683bf59ad2f1634f8dcad9c8c8d532515c025321106"}
Sep 17 10:59:57.922501 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:57.922459    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:58.071695 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:58.071652    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:58.270553 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:58.270471    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:58.438238 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:58.438198    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 10:59:58.550740 ip-10-0-139-204 kubenswrapper[2562]: I0917 10:59:58.550706    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:00.475950 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:00.475857    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n598x" event={"ID":"3a3cc953-5ff2-42ca-a53f-ae05c57581e3","Type":"ContainerStarted","Data":"318185607bc1ebbfdc698f68aa57d427df3237528f588597b6dee08731407df8"}
Sep 17 11:00:00.497707 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:00.497654    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n598x" podStartSLOduration=1.9360497479999998 podStartE2EDuration="6.497637686s" totalImagesPullingTime="4.561587938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 10:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:00:00.495228215 +0000 UTC m=+394.593273780" watchObservedRunningTime="2026-09-17 11:00:00.497637686 +0000 UTC m=+394.595683253"
Sep 17 11:00:04.809668 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:04.809634    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:04.880653 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:04.880616    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 11:00:04.880854 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:04.880678    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 11:00:04.885596 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:04.885569    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 11:00:05.164619 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:05.164532    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:05.502655 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:05.502575    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54cc58f9b9-28cjv"
Sep 17 11:00:05.940555 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:05.940519    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:06.237185 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:06.237105    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:06.538310 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:06.538283    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:09.074846 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:09.074801    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:09.289254 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:09.289224    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:12.287254 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:12.287219    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:12.434186 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:12.434155    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:17.625861 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:17.625830    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:17.733654 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:17.733613    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:18.044482 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:18.044451    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:18.161544 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:18.161504    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:18.479777 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:18.479691    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:18.587704 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:18.587669    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:23.868999 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:23.868964    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:23.967247 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:23.967210    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:28.612511 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:28.612472    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:28.711767 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:28.711731    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:36.953327 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:36.953289    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:37.590832 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:37.590796    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:42.872169 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:42.872129    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:43.027298 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:43.027255    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:43.345140 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:43.345102    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:00:43.366064 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:43.366027    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"
Sep 17 11:00:43.369671 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:43.369650    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 11:00:43.374708 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:43.374683    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:00:43.393718 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:43.393696    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:00:43.557641 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:43.557605    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvnh2\" (UniqueName: \"kubernetes.io/projected/b90e80c8-9401-4712-993c-f72cf2d92c5f-kube-api-access-jvnh2\") pod \"limitador-limitador-84994f7ffc-dr8cp\" (UID: \"b90e80c8-9401-4712-993c-f72cf2d92c5f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"
Sep 17 11:00:43.557842 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:43.557685    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b90e80c8-9401-4712-993c-f72cf2d92c5f-config-file\") pod \"limitador-limitador-84994f7ffc-dr8cp\" (UID: \"b90e80c8-9401-4712-993c-f72cf2d92c5f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"
Sep 17 11:00:43.658812 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:43.658716    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b90e80c8-9401-4712-993c-f72cf2d92c5f-config-file\") pod \"limitador-limitador-84994f7ffc-dr8cp\" (UID: \"b90e80c8-9401-4712-993c-f72cf2d92c5f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"
Sep 17 11:00:43.658812 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:43.658783    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvnh2\" (UniqueName: \"kubernetes.io/projected/b90e80c8-9401-4712-993c-f72cf2d92c5f-kube-api-access-jvnh2\") pod \"limitador-limitador-84994f7ffc-dr8cp\" (UID: \"b90e80c8-9401-4712-993c-f72cf2d92c5f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"
Sep 17 11:00:43.659423 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:43.659398    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b90e80c8-9401-4712-993c-f72cf2d92c5f-config-file\") pod \"limitador-limitador-84994f7ffc-dr8cp\" (UID: \"b90e80c8-9401-4712-993c-f72cf2d92c5f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"
Sep 17 11:00:43.672313 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:43.672286    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvnh2\" (UniqueName: \"kubernetes.io/projected/b90e80c8-9401-4712-993c-f72cf2d92c5f-kube-api-access-jvnh2\") pod \"limitador-limitador-84994f7ffc-dr8cp\" (UID: \"b90e80c8-9401-4712-993c-f72cf2d92c5f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"
Sep 17 11:00:43.677123 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:43.677100    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"
Sep 17 11:00:43.987104 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:43.987073    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:00:43.989701 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:00:43.989673    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90e80c8_9401_4712_993c_f72cf2d92c5f.slice/crio-a346acb884c8a3eb4100d2e4bab3d6ba35e1e8340c61d22943e1142d2625e1c0 WatchSource:0}: Error finding container a346acb884c8a3eb4100d2e4bab3d6ba35e1e8340c61d22943e1142d2625e1c0: Status 404 returned error can't find the container with id a346acb884c8a3eb4100d2e4bab3d6ba35e1e8340c61d22943e1142d2625e1c0
Sep 17 11:00:44.127798 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:44.127767    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:44.247598 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:44.247514    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:44.407853 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:44.407803    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-9dds9"]
Sep 17 11:00:44.412384 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:44.412368    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9dds9"
Sep 17 11:00:44.417028 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:44.417003    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-f92d9\""
Sep 17 11:00:44.425960 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:44.425935    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9dds9"]
Sep 17 11:00:44.567431 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:44.567395    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4l6v\" (UniqueName: \"kubernetes.io/projected/fb5682b8-0c76-47e3-9a80-cf0981e18f17-kube-api-access-w4l6v\") pod \"authorino-59c974bd48-9dds9\" (UID: \"fb5682b8-0c76-47e3-9a80-cf0981e18f17\") " pod="kuadrant-system/authorino-59c974bd48-9dds9"
Sep 17 11:00:44.668480 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:44.668443    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4l6v\" (UniqueName: \"kubernetes.io/projected/fb5682b8-0c76-47e3-9a80-cf0981e18f17-kube-api-access-w4l6v\") pod \"authorino-59c974bd48-9dds9\" (UID: \"fb5682b8-0c76-47e3-9a80-cf0981e18f17\") " pod="kuadrant-system/authorino-59c974bd48-9dds9"
Sep 17 11:00:44.678887 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:44.678859    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4l6v\" (UniqueName: \"kubernetes.io/projected/fb5682b8-0c76-47e3-9a80-cf0981e18f17-kube-api-access-w4l6v\") pod \"authorino-59c974bd48-9dds9\" (UID: \"fb5682b8-0c76-47e3-9a80-cf0981e18f17\") " pod="kuadrant-system/authorino-59c974bd48-9dds9"
Sep 17 11:00:44.684148 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:44.684120    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-dr8cp" event={"ID":"b90e80c8-9401-4712-993c-f72cf2d92c5f","Type":"ContainerStarted","Data":"a346acb884c8a3eb4100d2e4bab3d6ba35e1e8340c61d22943e1142d2625e1c0"}
Sep 17 11:00:44.722351 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:44.722315    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9dds9"
Sep 17 11:00:45.008942 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:45.008867    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9dds9"]
Sep 17 11:00:45.578676 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:45.578639    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:45.690868 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:45.690834    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9dds9" event={"ID":"fb5682b8-0c76-47e3-9a80-cf0981e18f17","Type":"ContainerStarted","Data":"37d9aa8861c276a031424e94915d1f2a9ba4a739ce70779b6fe7f3c1d8383a7d"}
Sep 17 11:00:45.716091 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:45.716051    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:45.935853 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:45.935766    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:46.056653 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:46.056615    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:00:46.699470 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:46.699422    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-dr8cp" event={"ID":"b90e80c8-9401-4712-993c-f72cf2d92c5f","Type":"ContainerStarted","Data":"439a722810e0de9f9a2642c9f101ad5f56bafa7a5966263021e3f2b224485282"}
Sep 17 11:00:46.699639 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:46.699514    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"
Sep 17 11:00:46.720603 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:46.720550    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-dr8cp" podStartSLOduration=1.14811924 podStartE2EDuration="3.720534782s" totalImagesPullingTime="2.572415542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:00:46.718368303 +0000 UTC m=+440.816413869" watchObservedRunningTime="2026-09-17 11:00:46.720534782 +0000 UTC m=+440.818580348"
Sep 17 11:00:48.709955 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:48.709918    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9dds9" event={"ID":"fb5682b8-0c76-47e3-9a80-cf0981e18f17","Type":"ContainerStarted","Data":"5ed1dc2ac19e3ac925e0ccf34f05236a46e38ee3671dce2f4c5f64c53eb14f87"}
Sep 17 11:00:48.732019 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:48.731968    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-9dds9" podStartSLOduration=2.062236113 podStartE2EDuration="4.731952509s" totalImagesPullingTime="2.669716396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:00:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:00:48.7299285 +0000 UTC m=+442.827974065" watchObservedRunningTime="2026-09-17 11:00:48.731952509 +0000 UTC m=+442.829998075"
Sep 17 11:00:57.705925 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:00:57.705893    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"
Sep 17 11:01:03.938937 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:03.938901    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:01:04.095989 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:04.095960    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:01:04.455634 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:04.455599    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:01:17.332922 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:17.332891    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:01:17.517977 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:17.517948    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:01:19.694498 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:19.694462    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9dds9"]
Sep 17 11:01:19.694871 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:19.694637    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-9dds9" podUID="fb5682b8-0c76-47e3-9a80-cf0981e18f17" containerName="authorino" containerID="cri-o://5ed1dc2ac19e3ac925e0ccf34f05236a46e38ee3671dce2f4c5f64c53eb14f87" gracePeriod=30
Sep 17 11:01:19.857711 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:19.857675    2562 generic.go:356] "Generic (PLEG): container finished" podID="fb5682b8-0c76-47e3-9a80-cf0981e18f17" containerID="5ed1dc2ac19e3ac925e0ccf34f05236a46e38ee3671dce2f4c5f64c53eb14f87" exitCode=0
Sep 17 11:01:19.857909 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:19.857755    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9dds9" event={"ID":"fb5682b8-0c76-47e3-9a80-cf0981e18f17","Type":"ContainerDied","Data":"5ed1dc2ac19e3ac925e0ccf34f05236a46e38ee3671dce2f4c5f64c53eb14f87"}
Sep 17 11:01:19.947281 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:19.947229    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9dds9"
Sep 17 11:01:20.081028 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:20.080997    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4l6v\" (UniqueName: \"kubernetes.io/projected/fb5682b8-0c76-47e3-9a80-cf0981e18f17-kube-api-access-w4l6v\") pod \"fb5682b8-0c76-47e3-9a80-cf0981e18f17\" (UID: \"fb5682b8-0c76-47e3-9a80-cf0981e18f17\") "
Sep 17 11:01:20.083196 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:20.083172    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5682b8-0c76-47e3-9a80-cf0981e18f17-kube-api-access-w4l6v" (OuterVolumeSpecName: "kube-api-access-w4l6v") pod "fb5682b8-0c76-47e3-9a80-cf0981e18f17" (UID: "fb5682b8-0c76-47e3-9a80-cf0981e18f17"). InnerVolumeSpecName "kube-api-access-w4l6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:01:20.182079 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:20.182043    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4l6v\" (UniqueName: \"kubernetes.io/projected/fb5682b8-0c76-47e3-9a80-cf0981e18f17-kube-api-access-w4l6v\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:01:20.863864 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:20.863832    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9dds9"
Sep 17 11:01:20.863864 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:20.863855    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9dds9" event={"ID":"fb5682b8-0c76-47e3-9a80-cf0981e18f17","Type":"ContainerDied","Data":"37d9aa8861c276a031424e94915d1f2a9ba4a739ce70779b6fe7f3c1d8383a7d"}
Sep 17 11:01:20.864335 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:20.863910    2562 scope.go:117] "RemoveContainer" containerID="5ed1dc2ac19e3ac925e0ccf34f05236a46e38ee3671dce2f4c5f64c53eb14f87"
Sep 17 11:01:20.910746 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:20.910716    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9dds9"]
Sep 17 11:01:20.924236 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:20.924204    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9dds9"]
Sep 17 11:01:22.455648 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:22.455615    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb5682b8-0c76-47e3-9a80-cf0981e18f17" path="/var/lib/kubelet/pods/fb5682b8-0c76-47e3-9a80-cf0981e18f17/volumes"
Sep 17 11:01:25.462083 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:25.462046    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:01:25.617527 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:25.617492    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:01:25.831994 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:25.831960    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:01:28.934949 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:28.934919    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:01:29.159664 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:29.159631    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:01:44.743265 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:44.743230    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:01:55.845441 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:55.845401    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:01:55.963097 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:01:55.963060    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:02:02.160944 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:02.160901    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:02:02.386735 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:02.386693    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:02:04.529843 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:04.529783    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:02:04.702709 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:04.702678    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:02:04.936347 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:04.936264    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:02:05.151801 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:05.151765    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:02:05.224397 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:05.224307    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:02:14.255070 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:14.255027    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:02:14.500545 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:14.500511    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:02:21.066366 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:21.066272    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:02:24.168411 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:24.168371    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:02:24.538761 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:24.538726    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:02:31.378656 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:31.378620    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:02:31.736013 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:31.735937    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:02:42.901969 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:42.901935    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:02:43.918456 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:43.918418    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:02:52.406309 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.406278    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"]
Sep 17 11:02:52.406682 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.406654    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5682b8-0c76-47e3-9a80-cf0981e18f17" containerName="authorino"
Sep 17 11:02:52.406682 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.406665    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5682b8-0c76-47e3-9a80-cf0981e18f17" containerName="authorino"
Sep 17 11:02:52.406771 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.406736    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5682b8-0c76-47e3-9a80-cf0981e18f17" containerName="authorino"
Sep 17 11:02:52.409644 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.409628    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.412498 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.412472    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 17 11:02:52.412643 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.412601    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 17 11:02:52.413848 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.413825    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4xkgq\""
Sep 17 11:02:52.414008 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.413886    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 17 11:02:52.422387 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.422362    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"]
Sep 17 11:02:52.443596 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.443554    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:02:52.493933 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.493898    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecff354-fce1-4221-85ca-b2a98adf0206-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p2bh9\" (UID: \"6ecff354-fce1-4221-85ca-b2a98adf0206\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.494100 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.493953    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecff354-fce1-4221-85ca-b2a98adf0206-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p2bh9\" (UID: \"6ecff354-fce1-4221-85ca-b2a98adf0206\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.494100 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.494008    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecff354-fce1-4221-85ca-b2a98adf0206-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p2bh9\" (UID: \"6ecff354-fce1-4221-85ca-b2a98adf0206\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.494100 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.494027    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn7df\" (UniqueName: \"kubernetes.io/projected/6ecff354-fce1-4221-85ca-b2a98adf0206-kube-api-access-zn7df\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p2bh9\" (UID: \"6ecff354-fce1-4221-85ca-b2a98adf0206\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.494100 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.494075    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecff354-fce1-4221-85ca-b2a98adf0206-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p2bh9\" (UID: \"6ecff354-fce1-4221-85ca-b2a98adf0206\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.494303 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.494115    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecff354-fce1-4221-85ca-b2a98adf0206-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p2bh9\" (UID: \"6ecff354-fce1-4221-85ca-b2a98adf0206\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.595225 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.595178    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecff354-fce1-4221-85ca-b2a98adf0206-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p2bh9\" (UID: \"6ecff354-fce1-4221-85ca-b2a98adf0206\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.595422 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.595243    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecff354-fce1-4221-85ca-b2a98adf0206-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p2bh9\" (UID: \"6ecff354-fce1-4221-85ca-b2a98adf0206\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.595422 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.595266    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecff354-fce1-4221-85ca-b2a98adf0206-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p2bh9\" (UID: \"6ecff354-fce1-4221-85ca-b2a98adf0206\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.595422 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.595286    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn7df\" (UniqueName: \"kubernetes.io/projected/6ecff354-fce1-4221-85ca-b2a98adf0206-kube-api-access-zn7df\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p2bh9\" (UID: \"6ecff354-fce1-4221-85ca-b2a98adf0206\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.595422 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.595307    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecff354-fce1-4221-85ca-b2a98adf0206-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p2bh9\" (UID: \"6ecff354-fce1-4221-85ca-b2a98adf0206\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.595422 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.595331    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecff354-fce1-4221-85ca-b2a98adf0206-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p2bh9\" (UID: \"6ecff354-fce1-4221-85ca-b2a98adf0206\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.595776 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.595739    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecff354-fce1-4221-85ca-b2a98adf0206-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p2bh9\" (UID: \"6ecff354-fce1-4221-85ca-b2a98adf0206\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.595776 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.595760    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecff354-fce1-4221-85ca-b2a98adf0206-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p2bh9\" (UID: \"6ecff354-fce1-4221-85ca-b2a98adf0206\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.596006 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.595809    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecff354-fce1-4221-85ca-b2a98adf0206-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p2bh9\" (UID: \"6ecff354-fce1-4221-85ca-b2a98adf0206\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.597581 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.597562    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecff354-fce1-4221-85ca-b2a98adf0206-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p2bh9\" (UID: \"6ecff354-fce1-4221-85ca-b2a98adf0206\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.597859 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.597842    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecff354-fce1-4221-85ca-b2a98adf0206-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p2bh9\" (UID: \"6ecff354-fce1-4221-85ca-b2a98adf0206\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.604841 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.604803    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn7df\" (UniqueName: \"kubernetes.io/projected/6ecff354-fce1-4221-85ca-b2a98adf0206-kube-api-access-zn7df\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p2bh9\" (UID: \"6ecff354-fce1-4221-85ca-b2a98adf0206\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.721244 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.721151    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:02:52.995175 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:52.995145    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"]
Sep 17 11:02:52.996353 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:02:52.996328    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ecff354_fce1_4221_85ca_b2a98adf0206.slice/crio-108d4bc2667f96fd3fc93b9b0cad38b2866379339250e97adbd6c24a278950fa WatchSource:0}: Error finding container 108d4bc2667f96fd3fc93b9b0cad38b2866379339250e97adbd6c24a278950fa: Status 404 returned error can't find the container with id 108d4bc2667f96fd3fc93b9b0cad38b2866379339250e97adbd6c24a278950fa
Sep 17 11:02:53.313515 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:53.313467    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9" event={"ID":"6ecff354-fce1-4221-85ca-b2a98adf0206","Type":"ContainerStarted","Data":"108d4bc2667f96fd3fc93b9b0cad38b2866379339250e97adbd6c24a278950fa"}
Sep 17 11:02:53.718378 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:53.718293    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:02:54.320831 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:54.320783    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9" event={"ID":"6ecff354-fce1-4221-85ca-b2a98adf0206","Type":"ContainerStarted","Data":"18530e20378eea7d03669db2bceaecd75552d6b5fb53b319da6c9eb41e2a397a"}
Sep 17 11:02:54.342388 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:02:54.342338    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9" podStartSLOduration=1.33236735 podStartE2EDuration="2.342322881s" totalImagesPullingTime="1.009955531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:02:54.341383803 +0000 UTC m=+568.439429365" watchObservedRunningTime="2026-09-17 11:02:54.342322881 +0000 UTC m=+568.440368446"
Sep 17 11:03:01.263294 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:03:01.263253    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:03:02.721923 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:03:02.721882    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:03:02.722404 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:03:02.721950    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:03:02.734034 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:03:02.734007    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:03:02.927911 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:03:02.927876    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:03:03.375776 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:03:03.375730    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p2bh9"
Sep 17 11:03:18.256938 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:03:18.256899    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:03:20.224344 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:03:20.224314    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:03:26.380992 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:03:26.380965    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 11:03:26.382666 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:03:26.382640    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 11:03:30.595110 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:03:30.595074    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:03:30.693043 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:03:30.692932    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:03:34.738064 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:03:34.738027    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:03:40.653713 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:03:40.653676    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:03:42.916162 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:03:42.916127    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:04:23.639935 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:23.639890    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:04:25.703067 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:25.703037    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84cd4f5699-jgql7"]
Sep 17 11:04:25.706339 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:25.706318    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84cd4f5699-jgql7"
Sep 17 11:04:25.712892 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:25.712567    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-f92d9\""
Sep 17 11:04:25.712892 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:25.712579    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 11:04:25.724284 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:25.724256    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84cd4f5699-jgql7"]
Sep 17 11:04:25.827770 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:25.827735    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8h49\" (UniqueName: \"kubernetes.io/projected/8d3bf480-7b79-4675-a30b-25b87d770a43-kube-api-access-p8h49\") pod \"authorino-84cd4f5699-jgql7\" (UID: \"8d3bf480-7b79-4675-a30b-25b87d770a43\") " pod="kuadrant-system/authorino-84cd4f5699-jgql7"
Sep 17 11:04:25.827969 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:25.827799    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8d3bf480-7b79-4675-a30b-25b87d770a43-tls-cert\") pod \"authorino-84cd4f5699-jgql7\" (UID: \"8d3bf480-7b79-4675-a30b-25b87d770a43\") " pod="kuadrant-system/authorino-84cd4f5699-jgql7"
Sep 17 11:04:25.928691 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:25.928634    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8d3bf480-7b79-4675-a30b-25b87d770a43-tls-cert\") pod \"authorino-84cd4f5699-jgql7\" (UID: \"8d3bf480-7b79-4675-a30b-25b87d770a43\") " pod="kuadrant-system/authorino-84cd4f5699-jgql7"
Sep 17 11:04:25.928930 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:25.928751    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8h49\" (UniqueName: \"kubernetes.io/projected/8d3bf480-7b79-4675-a30b-25b87d770a43-kube-api-access-p8h49\") pod \"authorino-84cd4f5699-jgql7\" (UID: \"8d3bf480-7b79-4675-a30b-25b87d770a43\") " pod="kuadrant-system/authorino-84cd4f5699-jgql7"
Sep 17 11:04:25.931159 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:25.931139    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8d3bf480-7b79-4675-a30b-25b87d770a43-tls-cert\") pod \"authorino-84cd4f5699-jgql7\" (UID: \"8d3bf480-7b79-4675-a30b-25b87d770a43\") " pod="kuadrant-system/authorino-84cd4f5699-jgql7"
Sep 17 11:04:25.938460 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:25.938432    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8h49\" (UniqueName: \"kubernetes.io/projected/8d3bf480-7b79-4675-a30b-25b87d770a43-kube-api-access-p8h49\") pod \"authorino-84cd4f5699-jgql7\" (UID: \"8d3bf480-7b79-4675-a30b-25b87d770a43\") " pod="kuadrant-system/authorino-84cd4f5699-jgql7"
Sep 17 11:04:26.014859 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:26.014749    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84cd4f5699-jgql7"
Sep 17 11:04:26.285922 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:26.285871    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84cd4f5699-jgql7"]
Sep 17 11:04:26.288252 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:04:26.288225    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d3bf480_7b79_4675_a30b_25b87d770a43.slice/crio-11ae7f43a6ba9257394975860e0223be74741f528d455b5d378a82dda492bf5a WatchSource:0}: Error finding container 11ae7f43a6ba9257394975860e0223be74741f528d455b5d378a82dda492bf5a: Status 404 returned error can't find the container with id 11ae7f43a6ba9257394975860e0223be74741f528d455b5d378a82dda492bf5a
Sep 17 11:04:26.763839 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:26.763730    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84cd4f5699-jgql7" event={"ID":"8d3bf480-7b79-4675-a30b-25b87d770a43","Type":"ContainerStarted","Data":"11ae7f43a6ba9257394975860e0223be74741f528d455b5d378a82dda492bf5a"}
Sep 17 11:04:27.771045 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:27.771000    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84cd4f5699-jgql7" event={"ID":"8d3bf480-7b79-4675-a30b-25b87d770a43","Type":"ContainerStarted","Data":"106d619b0a0e9236584493574f525995b4cee32e8720562f9305637b211b8ff2"}
Sep 17 11:04:27.805729 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:27.805664    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-84cd4f5699-jgql7" podStartSLOduration=2.014115942 podStartE2EDuration="2.805647375s" totalImagesPullingTime="791.531433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:04:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:04:27.804850384 +0000 UTC m=+661.902895951" watchObservedRunningTime="2026-09-17 11:04:27.805647375 +0000 UTC m=+661.903692941"
Sep 17 11:04:28.265070 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:28.265031    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:04:28.643996 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:28.643965    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:04:28.863044 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:28.863002    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:04:35.738405 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:35.738373    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:04:46.141695 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:46.141640    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:04:46.266231 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:04:46.266203    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:11.863589 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:11.863541    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:12.241312 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:12.241218    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:13.464573 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:13.464533    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:13.564840 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:13.564786    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:14.654991 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:14.654951    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:14.752577 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:14.752540    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:32.342975 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:32.342875    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:32.537720 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:32.537639    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:33.005413 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:33.005380    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:33.254378 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:33.254337    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:33.396742 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:33.396696    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:33.567579 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:33.567534    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:35.231178 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:35.231136    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:36.086720 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:36.086691    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:36.369746 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:36.369658    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:36.581230 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:36.581195    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:37.087469 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:37.087428    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:37.404961 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:37.404875    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:37.743554 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:37.743466    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:38.108579 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:38.108547    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:38.238004 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:38.237971    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:38.500012 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:38.499920    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:39.298214 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:39.298180    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:39.391589 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:39.391556    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:39.493199 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:39.493164    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs"]
Sep 17 11:05:39.496787 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:39.496772    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs"
Sep 17 11:05:39.501649 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:39.501627    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d7b97f-dockercfg-fkj6f\""
Sep 17 11:05:39.501993 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:39.501639    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d7b97f\""
Sep 17 11:05:39.510401 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:39.510379    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs"]
Sep 17 11:05:39.578397 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:39.578314    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx448\" (UniqueName: \"kubernetes.io/projected/7b686a84-7754-4679-b61f-1f927c4fc965-kube-api-access-fx448\") pod \"payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs\" (UID: \"7b686a84-7754-4679-b61f-1f927c4fc965\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs"
Sep 17 11:05:39.578397 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:39.578350    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b686a84-7754-4679-b61f-1f927c4fc965-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs\" (UID: \"7b686a84-7754-4679-b61f-1f927c4fc965\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs"
Sep 17 11:05:39.679413 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:39.679381    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx448\" (UniqueName: \"kubernetes.io/projected/7b686a84-7754-4679-b61f-1f927c4fc965-kube-api-access-fx448\") pod \"payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs\" (UID: \"7b686a84-7754-4679-b61f-1f927c4fc965\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs"
Sep 17 11:05:39.679413 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:39.679415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b686a84-7754-4679-b61f-1f927c4fc965-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs\" (UID: \"7b686a84-7754-4679-b61f-1f927c4fc965\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs"
Sep 17 11:05:39.680185 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:39.680162    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b686a84-7754-4679-b61f-1f927c4fc965-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs\" (UID: \"7b686a84-7754-4679-b61f-1f927c4fc965\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs"
Sep 17 11:05:39.700423 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:39.700401    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx448\" (UniqueName: \"kubernetes.io/projected/7b686a84-7754-4679-b61f-1f927c4fc965-kube-api-access-fx448\") pod \"payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs\" (UID: \"7b686a84-7754-4679-b61f-1f927c4fc965\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs"
Sep 17 11:05:39.807174 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:39.807139    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs"
Sep 17 11:05:39.887098 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:39.887068    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:39.988171 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:39.988137    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:40.095718 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:40.095694    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs"]
Sep 17 11:05:40.097784 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:05:40.097756    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b686a84_7754_4679_b61f_1f927c4fc965.slice/crio-87a9930d5a1d512368f402d18ac132adc0186b4e8f9356cad45efda8c777fa1a WatchSource:0}: Error finding container 87a9930d5a1d512368f402d18ac132adc0186b4e8f9356cad45efda8c777fa1a: Status 404 returned error can't find the container with id 87a9930d5a1d512368f402d18ac132adc0186b4e8f9356cad45efda8c777fa1a
Sep 17 11:05:40.099775 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:40.099755    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 11:05:40.141653 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:40.141563    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs" event={"ID":"7b686a84-7754-4679-b61f-1f927c4fc965","Type":"ContainerStarted","Data":"87a9930d5a1d512368f402d18ac132adc0186b4e8f9356cad45efda8c777fa1a"}
Sep 17 11:05:40.415250 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:40.415164    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:40.632370 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:40.632340    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:40.746248 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:40.746169    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:40.792744 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:40.792710    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4"]
Sep 17 11:05:40.796803 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:40.796781    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4"
Sep 17 11:05:40.801244 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:40.801055    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-79a70a-dockercfg-74jfk\""
Sep 17 11:05:40.801244 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:40.801094    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-79a70a\""
Sep 17 11:05:40.823525 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:40.823464    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4"]
Sep 17 11:05:40.892104 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:40.892069    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13422c64-a518-44f5-9772-c39c50d5962e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4\" (UID: \"13422c64-a518-44f5-9772-c39c50d5962e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4"
Sep 17 11:05:40.892104 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:40.892103    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6b74\" (UniqueName: \"kubernetes.io/projected/13422c64-a518-44f5-9772-c39c50d5962e-kube-api-access-s6b74\") pod \"payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4\" (UID: \"13422c64-a518-44f5-9772-c39c50d5962e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4"
Sep 17 11:05:40.993469 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:40.993427    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13422c64-a518-44f5-9772-c39c50d5962e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4\" (UID: \"13422c64-a518-44f5-9772-c39c50d5962e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4"
Sep 17 11:05:40.993469 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:40.993464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6b74\" (UniqueName: \"kubernetes.io/projected/13422c64-a518-44f5-9772-c39c50d5962e-kube-api-access-s6b74\") pod \"payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4\" (UID: \"13422c64-a518-44f5-9772-c39c50d5962e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4"
Sep 17 11:05:40.994277 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:40.994251    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13422c64-a518-44f5-9772-c39c50d5962e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4\" (UID: \"13422c64-a518-44f5-9772-c39c50d5962e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4"
Sep 17 11:05:41.003869 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:41.003787    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6b74\" (UniqueName: \"kubernetes.io/projected/13422c64-a518-44f5-9772-c39c50d5962e-kube-api-access-s6b74\") pod \"payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4\" (UID: \"13422c64-a518-44f5-9772-c39c50d5962e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4"
Sep 17 11:05:41.111573 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:41.111536    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4"
Sep 17 11:05:41.363800 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:41.363756    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:41.441424 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:41.441392    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:41.639176 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:41.639091    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:41.694358 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:41.694326    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4"]
Sep 17 11:05:41.695117 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:05:41.695090    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13422c64_a518_44f5_9772_c39c50d5962e.slice/crio-406816b75c17facaf7e6d6ce93d39e078558bac3808dbfc46e8be7e6765acf94 WatchSource:0}: Error finding container 406816b75c17facaf7e6d6ce93d39e078558bac3808dbfc46e8be7e6765acf94: Status 404 returned error can't find the container with id 406816b75c17facaf7e6d6ce93d39e078558bac3808dbfc46e8be7e6765acf94
Sep 17 11:05:42.155183 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:42.155136    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs" event={"ID":"7b686a84-7754-4679-b61f-1f927c4fc965","Type":"ContainerStarted","Data":"6c2e580295d391cd727eda46e41b93663fc1ba873a12702c9ca66046955265f5"}
Sep 17 11:05:42.155387 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:42.155362    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs"
Sep 17 11:05:42.156980 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:42.156952    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4" event={"ID":"13422c64-a518-44f5-9772-c39c50d5962e","Type":"ContainerStarted","Data":"2ead94f3a16524d900c99ec94f246f2c2c620ca98b2f21be2156c8a45bb95957"}
Sep 17 11:05:42.157113 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:42.156991    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4" event={"ID":"13422c64-a518-44f5-9772-c39c50d5962e","Type":"ContainerStarted","Data":"406816b75c17facaf7e6d6ce93d39e078558bac3808dbfc46e8be7e6765acf94"}
Sep 17 11:05:42.157113 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:42.157068    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4"
Sep 17 11:05:42.202119 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:42.202061    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs" podStartSLOduration=1.841593848 podStartE2EDuration="3.202044264s" totalImagesPullingTime="1.360450416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:05:42.199451251 +0000 UTC m=+736.297496813" watchObservedRunningTime="2026-09-17 11:05:42.202044264 +0000 UTC m=+736.300089829"
Sep 17 11:05:42.269160 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:42.269101    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:42.390427 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:42.390394    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:42.740717 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:42.740660    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4" podStartSLOduration=2.740640646 podStartE2EDuration="2.740640646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:05:42.2463295 +0000 UTC m=+736.344375065" watchObservedRunningTime="2026-09-17 11:05:42.740640646 +0000 UTC m=+736.838686212"
Sep 17 11:05:42.741215 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:42.740969    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:05:42.878910 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:42.878871    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:44.195938 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:44.195903    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:44.981793 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:44.981734    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:45.106600 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:45.106565    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:45.216337 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:45.216306    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:45.369177 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:45.369124    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:52.291044 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:52.291002    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:52.481659 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:52.481626    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:52.590062 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:52.589955    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:52.703534 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:52.703497    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:05:53.165288 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:53.165258    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4"
Sep 17 11:05:53.165470 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:53.165333    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs"
Sep 17 11:05:57.636170 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:57.636126    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:05:59.997751 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:05:59.997717    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:00.323435 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:00.323403    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:00.469462 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:00.469427    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:01.956802 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:01.956765    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:03.191938 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:03.191907    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:04.274250 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:04.274217    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:06.411099 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:06.411064    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:06.515927 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:06.515890    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:06.668235 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:06.668158    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:06.827316 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:06.827283    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:07.024965 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:07.024890    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:07.170627 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:07.170576    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:07.376301 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:07.376269    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:07.768437 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:07.768355    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:07.946252 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:07.946214    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:08.074035 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:08.074006    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:11.243577 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:11.243542    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:11.370369 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:11.370331    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:11.803343 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:11.803307    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:13.204560 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:13.204527    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:14.554655 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.554614    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"]
Sep 17 11:06:14.559274 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.559249    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:06:14.563408 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.563381    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c10e0b-serving-cert\""
Sep 17 11:06:14.563569 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.563415    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 17 11:06:14.564750 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.564724    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 11:06:14.564922 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.564731    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 17 11:06:14.565096 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.565051    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gvplj\""
Sep 17 11:06:14.574407 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.574314    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"]
Sep 17 11:06:14.616504 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.616470    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:14.699307 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.699270    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:14.717517 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.717484    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq\" (UID: \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:06:14.717677 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.717523    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-metrics-certs\") pod \"maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq\" (UID: \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:06:14.717677 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.717548    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2lp2\" (UniqueName: \"kubernetes.io/projected/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-kube-api-access-s2lp2\") pod \"maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq\" (UID: \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:06:14.717677 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.717652    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-maas-api-tls\") pod \"maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq\" (UID: \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:06:14.819024 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.818938    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-maas-api-tls\") pod \"maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq\" (UID: \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:06:14.819024 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.819000    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq\" (UID: \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:06:14.819205 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.819030    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-metrics-certs\") pod \"maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq\" (UID: \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:06:14.819205 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.819055    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2lp2\" (UniqueName: \"kubernetes.io/projected/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-kube-api-access-s2lp2\") pod \"maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq\" (UID: \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:06:14.819205 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:06:14.819090    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-serving-cert: secret "maas-api-e2e-inf-b-c10e0b-serving-cert" not found
Sep 17 11:06:14.819205 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:06:14.819166    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-maas-api-tls podName:52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0 nodeName:}" failed. No retries permitted until 2026-09-17 11:06:15.319148304 +0000 UTC m=+769.417193853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-maas-api-tls") pod "maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq" (UID: "52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0") : secret "maas-api-e2e-inf-b-c10e0b-serving-cert" not found
Sep 17 11:06:14.819776 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.819745    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq\" (UID: \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:06:14.821459 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.821438    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-metrics-certs\") pod \"maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq\" (UID: \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:06:14.829353 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:14.829329    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2lp2\" (UniqueName: \"kubernetes.io/projected/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-kube-api-access-s2lp2\") pod \"maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq\" (UID: \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:06:15.324627 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.324593    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-maas-api-tls\") pod \"maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq\" (UID: \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:06:15.327147 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.327113    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-maas-api-tls\") pod \"maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq\" (UID: \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:06:15.404109 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.404079    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq"]
Sep 17 11:06:15.407900 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.407883    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq"
Sep 17 11:06:15.413229 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.413207    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c10e0b\""
Sep 17 11:06:15.471278 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.471244    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:06:15.476274 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.476249    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc"]
Sep 17 11:06:15.479978 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.479958    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq"]
Sep 17 11:06:15.480103 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.480065    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc"
Sep 17 11:06:15.483026 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.483008    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c10e0b-dockercfg-g8sdr\""
Sep 17 11:06:15.498239 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.498211    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc"]
Sep 17 11:06:15.526925 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.526788    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmdj8\" (UniqueName: \"kubernetes.io/projected/3212535a-e929-48ff-9cef-5d9e7eaf889b-kube-api-access-tmdj8\") pod \"payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq\" (UID: \"3212535a-e929-48ff-9cef-5d9e7eaf889b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq"
Sep 17 11:06:15.526925 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.526857    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3212535a-e929-48ff-9cef-5d9e7eaf889b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq\" (UID: \"3212535a-e929-48ff-9cef-5d9e7eaf889b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq"
Sep 17 11:06:15.628429 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.628335    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmdj8\" (UniqueName: \"kubernetes.io/projected/3212535a-e929-48ff-9cef-5d9e7eaf889b-kube-api-access-tmdj8\") pod \"payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq\" (UID: \"3212535a-e929-48ff-9cef-5d9e7eaf889b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq"
Sep 17 11:06:15.628429 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.628377    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3212535a-e929-48ff-9cef-5d9e7eaf889b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq\" (UID: \"3212535a-e929-48ff-9cef-5d9e7eaf889b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq"
Sep 17 11:06:15.628845 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.628430    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmrtv\" (UniqueName: \"kubernetes.io/projected/0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad-kube-api-access-mmrtv\") pod \"payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc\" (UID: \"0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc"
Sep 17 11:06:15.628845 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.628483    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc\" (UID: \"0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc"
Sep 17 11:06:15.629129 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.629110    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3212535a-e929-48ff-9cef-5d9e7eaf889b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq\" (UID: \"3212535a-e929-48ff-9cef-5d9e7eaf889b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq"
Sep 17 11:06:15.637992 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.637969    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmdj8\" (UniqueName: \"kubernetes.io/projected/3212535a-e929-48ff-9cef-5d9e7eaf889b-kube-api-access-tmdj8\") pod \"payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq\" (UID: \"3212535a-e929-48ff-9cef-5d9e7eaf889b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq"
Sep 17 11:06:15.718859 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.718791    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq"
Sep 17 11:06:15.729785 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.729755    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmrtv\" (UniqueName: \"kubernetes.io/projected/0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad-kube-api-access-mmrtv\") pod \"payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc\" (UID: \"0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc"
Sep 17 11:06:15.729939 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.729794    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc\" (UID: \"0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc"
Sep 17 11:06:15.730462 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.730441    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc\" (UID: \"0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc"
Sep 17 11:06:15.744364 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.744335    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmrtv\" (UniqueName: \"kubernetes.io/projected/0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad-kube-api-access-mmrtv\") pod \"payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc\" (UID: \"0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc"
Sep 17 11:06:15.797211 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.797177    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"]
Sep 17 11:06:15.797825 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:06:15.797785    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52fa6f4b_6663_4ea7_a21c_efb2a32ba3c0.slice/crio-a3f588662c189d28a24f784287a8a37cbb10cc1ee7e2b11a327aaada4e0536b8 WatchSource:0}: Error finding container a3f588662c189d28a24f784287a8a37cbb10cc1ee7e2b11a327aaada4e0536b8: Status 404 returned error can't find the container with id a3f588662c189d28a24f784287a8a37cbb10cc1ee7e2b11a327aaada4e0536b8
Sep 17 11:06:15.816984 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.816949    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc"
Sep 17 11:06:15.987407 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.987320    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:15.998458 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:15.998434    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq"]
Sep 17 11:06:15.999723 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:06:15.999697    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3212535a_e929_48ff_9cef_5d9e7eaf889b.slice/crio-5378c449565dafd897ac3528377375c6ada4d7591dd6772219cb9a75981bc8cf WatchSource:0}: Error finding container 5378c449565dafd897ac3528377375c6ada4d7591dd6772219cb9a75981bc8cf: Status 404 returned error can't find the container with id 5378c449565dafd897ac3528377375c6ada4d7591dd6772219cb9a75981bc8cf
Sep 17 11:06:16.092539 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:16.092504    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:16.107143 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:16.107112    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc"]
Sep 17 11:06:16.108758 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:06:16.108736    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c86dcfb_ff1e_4063_bd5e_e12d0e8a75ad.slice/crio-29076d80b8b70626b8a2406a4403ded0cedaefcbc325fb2b245826620f9a92ff WatchSource:0}: Error finding container 29076d80b8b70626b8a2406a4403ded0cedaefcbc325fb2b245826620f9a92ff: Status 404 returned error can't find the container with id 29076d80b8b70626b8a2406a4403ded0cedaefcbc325fb2b245826620f9a92ff
Sep 17 11:06:16.336959 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:16.336908    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc" event={"ID":"0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad","Type":"ContainerStarted","Data":"1d77635fa7adf8c3c66852e7cf6bc0fd792949a0e8382cb23316d666f70d8350"}
Sep 17 11:06:16.337139 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:16.336972    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc" event={"ID":"0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad","Type":"ContainerStarted","Data":"29076d80b8b70626b8a2406a4403ded0cedaefcbc325fb2b245826620f9a92ff"}
Sep 17 11:06:16.337139 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:16.337088    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc"
Sep 17 11:06:16.338678 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:16.338657    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq" event={"ID":"3212535a-e929-48ff-9cef-5d9e7eaf889b","Type":"ContainerStarted","Data":"e95c66f3e1ab6e145399cd0e0562d3071aa6e927d5be8ca043958c5e5c3ed10f"}
Sep 17 11:06:16.338809 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:16.338693    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq" event={"ID":"3212535a-e929-48ff-9cef-5d9e7eaf889b","Type":"ContainerStarted","Data":"5378c449565dafd897ac3528377375c6ada4d7591dd6772219cb9a75981bc8cf"}
Sep 17 11:06:16.338809 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:16.338715    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq"
Sep 17 11:06:16.339974 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:16.339953    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq" event={"ID":"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0","Type":"ContainerStarted","Data":"a3f588662c189d28a24f784287a8a37cbb10cc1ee7e2b11a327aaada4e0536b8"}
Sep 17 11:06:16.366980 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:16.366907    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc" podStartSLOduration=1.366886915 podStartE2EDuration="1.366886915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:06:16.366138841 +0000 UTC m=+770.464184412" watchObservedRunningTime="2026-09-17 11:06:16.366886915 +0000 UTC m=+770.464932481"
Sep 17 11:06:16.414796 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:16.409543    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq" podStartSLOduration=1.4095217 podStartE2EDuration="1.4095217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:06:16.406092807 +0000 UTC m=+770.504138366" watchObservedRunningTime="2026-09-17 11:06:16.4095217 +0000 UTC m=+770.507567267"
Sep 17 11:06:17.038439 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:17.038407    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:19.359943 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:19.359811    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq" event={"ID":"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0","Type":"ContainerStarted","Data":"8f17d107aaf18f316183f09a20f5bd25be847bb4243760829a710b23a17579c7"}
Sep 17 11:06:19.359943 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:19.359913    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:06:19.407125 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:19.407073    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq" podStartSLOduration=2.162101502 podStartE2EDuration="5.407054471s" totalImagesPullingTime="3.244952969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:06:19.405759377 +0000 UTC m=+773.503804943" watchObservedRunningTime="2026-09-17 11:06:19.407054471 +0000 UTC m=+773.505100037"
Sep 17 11:06:20.525591 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:20.525554    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:06:25.370309 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:25.370279    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:06:26.382320 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:26.382283    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:27.349000 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:27.348958    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc"
Sep 17 11:06:27.349170 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:27.349038    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq"
Sep 17 11:06:35.295799 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:35.295764    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:37.674001 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:37.673965    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:38.470877 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:38.470842    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:39.193680 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:39.193640    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:40.228507 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:40.228467    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:06:40.252180 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:40.252126    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:41.299744 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:41.299710    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:58.520622 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.520543    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"]
Sep 17 11:06:58.524441 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.524418    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.529514 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.529490    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-79a70a\"/\"openshift-service-ca.crt\""
Sep 17 11:06:58.529648 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.529491    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-79a70a\"/\"default-dockercfg-djk5z\""
Sep 17 11:06:58.529648 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.529491    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-79a70a\"/\"test-model-79a70a-kserve-self-signed-certs\""
Sep 17 11:06:58.529648 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.529496    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-79a70a\"/\"kube-root-ca.crt\""
Sep 17 11:06:58.542228 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.542197    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"]
Sep 17 11:06:58.622432 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.622397    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:06:58.625142 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.625124    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnjwh\" (UniqueName: \"kubernetes.io/projected/53c60f88-cc3f-4e03-a6cd-4913652be060-kube-api-access-wnjwh\") pod \"test-model-79a70a-kserve-694494b495-5hh88\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") " pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.625209 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.625151    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-dshm\") pod \"test-model-79a70a-kserve-694494b495-5hh88\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") " pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.625209 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.625169    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53c60f88-cc3f-4e03-a6cd-4913652be060-tls-certs\") pod \"test-model-79a70a-kserve-694494b495-5hh88\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") " pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.625328 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.625304    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-tmp-dir\") pod \"test-model-79a70a-kserve-694494b495-5hh88\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") " pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.625387 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.625346    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-model-cache\") pod \"test-model-79a70a-kserve-694494b495-5hh88\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") " pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.625387 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.625380    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-home\") pod \"test-model-79a70a-kserve-694494b495-5hh88\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") " pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.726804 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.726767    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-tmp-dir\") pod \"test-model-79a70a-kserve-694494b495-5hh88\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") " pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.726804 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.726802    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-model-cache\") pod \"test-model-79a70a-kserve-694494b495-5hh88\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") " pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.727077 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.726846    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-home\") pod \"test-model-79a70a-kserve-694494b495-5hh88\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") " pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.727077 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.726926    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnjwh\" (UniqueName: \"kubernetes.io/projected/53c60f88-cc3f-4e03-a6cd-4913652be060-kube-api-access-wnjwh\") pod \"test-model-79a70a-kserve-694494b495-5hh88\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") " pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.727077 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.726953    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-dshm\") pod \"test-model-79a70a-kserve-694494b495-5hh88\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") " pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.727077 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.726975    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53c60f88-cc3f-4e03-a6cd-4913652be060-tls-certs\") pod \"test-model-79a70a-kserve-694494b495-5hh88\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") " pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.727295 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.727201    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-model-cache\") pod \"test-model-79a70a-kserve-694494b495-5hh88\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") " pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.727295 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.727265    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-tmp-dir\") pod \"test-model-79a70a-kserve-694494b495-5hh88\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") " pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.727295 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.727287    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-home\") pod \"test-model-79a70a-kserve-694494b495-5hh88\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") " pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.729188 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.729162    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-dshm\") pod \"test-model-79a70a-kserve-694494b495-5hh88\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") " pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.729591 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.729568    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53c60f88-cc3f-4e03-a6cd-4913652be060-tls-certs\") pod \"test-model-79a70a-kserve-694494b495-5hh88\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") " pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.735984 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.735961    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnjwh\" (UniqueName: \"kubernetes.io/projected/53c60f88-cc3f-4e03-a6cd-4913652be060-kube-api-access-wnjwh\") pod \"test-model-79a70a-kserve-694494b495-5hh88\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") " pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:58.833991 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:58.833955    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:06:59.107842 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:59.107783    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"]
Sep 17 11:06:59.109794 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:06:59.109770    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53c60f88_cc3f_4e03_a6cd_4913652be060.slice/crio-d814ae357253d1ae69dcad5f4ae19895e517c970843a93151f1f71fda900e170 WatchSource:0}: Error finding container d814ae357253d1ae69dcad5f4ae19895e517c970843a93151f1f71fda900e170: Status 404 returned error can't find the container with id d814ae357253d1ae69dcad5f4ae19895e517c970843a93151f1f71fda900e170
Sep 17 11:06:59.568640 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:59.568605    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88" event={"ID":"53c60f88-cc3f-4e03-a6cd-4913652be060","Type":"ContainerStarted","Data":"4d35f80aff1a2f642ed49f50a4697263ea63804b19abe36bb09e1e512d0106de"}
Sep 17 11:06:59.568640 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:59.568647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88" event={"ID":"53c60f88-cc3f-4e03-a6cd-4913652be060","Type":"ContainerStarted","Data":"d814ae357253d1ae69dcad5f4ae19895e517c970843a93151f1f71fda900e170"}
Sep 17 11:06:59.588273 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:06:59.588225    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88" podStartSLOduration=1.588206899 podStartE2EDuration="1.588206899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:06:59.586953632 +0000 UTC m=+813.684999198" watchObservedRunningTime="2026-09-17 11:06:59.588206899 +0000 UTC m=+813.686252466"
Sep 17 11:07:03.635348 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:03.635297    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:07:08.834831 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:08.834775    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:07:08.835315 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:08.834859    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:07:08.847523 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:08.847495    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:07:09.632751 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:09.632716    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:07:14.865406 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:14.865374    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:07:17.235281 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.235244    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:07:17.619844 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.619795    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"]
Sep 17 11:07:17.623526 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.623510    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.626655 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.626633    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d7b97f\"/\"kube-root-ca.crt\""
Sep 17 11:07:17.627840 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.627806    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d7b97f\"/\"openshift-service-ca.crt\""
Sep 17 11:07:17.627954 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.627809    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d7b97f\"/\"premium-w0-d7b97f-kserve-self-signed-certs\""
Sep 17 11:07:17.628011 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.627995    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d7b97f\"/\"default-dockercfg-v7mwd\""
Sep 17 11:07:17.636555 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.636530    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"]
Sep 17 11:07:17.696014 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.695981    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-dshm\") pod \"premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") " pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.696014 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.696013    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-model-cache\") pod \"premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") " pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.696216 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.696040    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-home\") pod \"premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") " pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.696216 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.696153    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kx9s\" (UniqueName: \"kubernetes.io/projected/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-kube-api-access-9kx9s\") pod \"premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") " pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.696216 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.696183    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-tmp-dir\") pod \"premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") " pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.696315 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.696249    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-tls-certs\") pod \"premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") " pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.749997 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.749962    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:07:17.796693 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.796656    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-dshm\") pod \"premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") " pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.796693 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.796694    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-model-cache\") pod \"premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") " pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.796912 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.796808    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-home\") pod \"premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") " pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.796954 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.796927    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kx9s\" (UniqueName: \"kubernetes.io/projected/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-kube-api-access-9kx9s\") pod \"premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") " pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.796994 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.796956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-tmp-dir\") pod \"premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") " pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.797113 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.797082    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-tls-certs\") pod \"premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") " pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.797203 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.797156    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-model-cache\") pod \"premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") " pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.797203 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.797184    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-home\") pod \"premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") " pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.797306 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.797239    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-tmp-dir\") pod \"premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") " pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.799001 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.798980    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-dshm\") pod \"premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") " pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.799326 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.799310    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-tls-certs\") pod \"premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") " pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.806860 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.806829    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kx9s\" (UniqueName: \"kubernetes.io/projected/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-kube-api-access-9kx9s\") pod \"premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") " pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:17.933323 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:17.933237    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:18.208103 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:18.208079    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"]
Sep 17 11:07:18.210722 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:07:18.210692    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd4df3cf_5c57_4753_a2e4_a4bbb3b22fd1.slice/crio-a7fab5169de79aab5d5d47d4b1beecf7845b5e8f893c81608799acf444b307df WatchSource:0}: Error finding container a7fab5169de79aab5d5d47d4b1beecf7845b5e8f893c81608799acf444b307df: Status 404 returned error can't find the container with id a7fab5169de79aab5d5d47d4b1beecf7845b5e8f893c81608799acf444b307df
Sep 17 11:07:18.668024 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:18.667979    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh" event={"ID":"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1","Type":"ContainerStarted","Data":"172cf5f0a0add42d5cb95b5514e3c8cf37b15192b4e456bec17e1fa58e60dac4"}
Sep 17 11:07:18.668024 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:18.668032    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh" event={"ID":"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1","Type":"ContainerStarted","Data":"a7fab5169de79aab5d5d47d4b1beecf7845b5e8f893c81608799acf444b307df"}
Sep 17 11:07:18.687447 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:18.687394    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh" podStartSLOduration=1.687378848 podStartE2EDuration="1.687378848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:07:18.685482735 +0000 UTC m=+832.783528325" watchObservedRunningTime="2026-09-17 11:07:18.687378848 +0000 UTC m=+832.785424414"
Sep 17 11:07:27.933936 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:27.933895    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:27.934370 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:27.933951    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:27.946562 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:27.946536    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:28.744433 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:28.744399    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:07:31.320445 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:31.320412    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:07:32.881223 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:32.881192    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:07:47.138927 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:07:47.138892    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:08:22.623528 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:22.623495    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:08:26.417689 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:26.417653    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 11:08:26.418477 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:26.418457    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 11:08:39.067332 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:39.067288    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9057-baa383"]
Sep 17 11:08:39.070929 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:39.070909    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9057-baa383"
Sep 17 11:08:39.074473 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:39.074452    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bnkhs\""
Sep 17 11:08:39.083358 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:39.083335    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9057-baa383"]
Sep 17 11:08:39.186719 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:39.186680    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlg4d\" (UniqueName: \"kubernetes.io/projected/923d5e8e-2b94-44bb-9aca-c68d43e4e468-kube-api-access-vlg4d\") pod \"test-curl-9057-baa383\" (UID: \"923d5e8e-2b94-44bb-9aca-c68d43e4e468\") " pod="opendatahub/test-curl-9057-baa383"
Sep 17 11:08:39.287654 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:39.287610    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlg4d\" (UniqueName: \"kubernetes.io/projected/923d5e8e-2b94-44bb-9aca-c68d43e4e468-kube-api-access-vlg4d\") pod \"test-curl-9057-baa383\" (UID: \"923d5e8e-2b94-44bb-9aca-c68d43e4e468\") " pod="opendatahub/test-curl-9057-baa383"
Sep 17 11:08:39.298611 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:39.298580    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlg4d\" (UniqueName: \"kubernetes.io/projected/923d5e8e-2b94-44bb-9aca-c68d43e4e468-kube-api-access-vlg4d\") pod \"test-curl-9057-baa383\" (UID: \"923d5e8e-2b94-44bb-9aca-c68d43e4e468\") " pod="opendatahub/test-curl-9057-baa383"
Sep 17 11:08:39.380886 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:39.380764    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9057-baa383"
Sep 17 11:08:39.652642 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:39.652620    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9057-baa383"]
Sep 17 11:08:39.654945 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:08:39.654922    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod923d5e8e_2b94_44bb_9aca_c68d43e4e468.slice/crio-bb964312b3e457285256b470b430b28ffaa6e96e9579973c6b67615c145ed7a4 WatchSource:0}: Error finding container bb964312b3e457285256b470b430b28ffaa6e96e9579973c6b67615c145ed7a4: Status 404 returned error can't find the container with id bb964312b3e457285256b470b430b28ffaa6e96e9579973c6b67615c145ed7a4
Sep 17 11:08:40.098620 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:40.098579    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9057-baa383" event={"ID":"923d5e8e-2b94-44bb-9aca-c68d43e4e468","Type":"ContainerStarted","Data":"bb964312b3e457285256b470b430b28ffaa6e96e9579973c6b67615c145ed7a4"}
Sep 17 11:08:42.111730 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:42.111691    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9057-baa383" event={"ID":"923d5e8e-2b94-44bb-9aca-c68d43e4e468","Type":"ContainerStarted","Data":"1d507faa1bc0a1f7011979b1dacb4c8f17b25b01bca103366b83394ac9832aa8"}
Sep 17 11:08:42.141319 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:42.141247    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9057-baa383" podStartSLOduration=1.080420782 podStartE2EDuration="3.141231301s" totalImagesPullingTime="2.060810519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:08:42.140642638 +0000 UTC m=+916.238688203" watchObservedRunningTime="2026-09-17 11:08:42.141231301 +0000 UTC m=+916.239276867"
Sep 17 11:08:42.798957 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:42.798925    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9057-baa383"]
Sep 17 11:08:42.809109 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:42.809080    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9057-baa383"]
Sep 17 11:08:42.941426 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:42.941348    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9057-b2a70d"]
Sep 17 11:08:42.941786 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:42.941774    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="923d5e8e-2b94-44bb-9aca-c68d43e4e468" containerName="test-curl-9057-baa383"
Sep 17 11:08:42.941859 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:42.941788    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="923d5e8e-2b94-44bb-9aca-c68d43e4e468" containerName="test-curl-9057-baa383"
Sep 17 11:08:42.941908 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:42.941869    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="923d5e8e-2b94-44bb-9aca-c68d43e4e468" containerName="test-curl-9057-baa383"
Sep 17 11:08:42.945018 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:42.945004    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9057-b2a70d"
Sep 17 11:08:42.953732 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:42.953706    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9057-b2a70d"]
Sep 17 11:08:43.021136 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:43.021102    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6j8x\" (UniqueName: \"kubernetes.io/projected/3647c8cf-e471-42cf-ae45-02eea81f7c06-kube-api-access-p6j8x\") pod \"test-curl-9057-b2a70d\" (UID: \"3647c8cf-e471-42cf-ae45-02eea81f7c06\") " pod="opendatahub/test-curl-9057-b2a70d"
Sep 17 11:08:43.116648 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:08:43.116618    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9057-baa383" container="test-curl-9057-baa383"
Sep 17 11:08:43.121533 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:43.121503    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6j8x\" (UniqueName: \"kubernetes.io/projected/3647c8cf-e471-42cf-ae45-02eea81f7c06-kube-api-access-p6j8x\") pod \"test-curl-9057-b2a70d\" (UID: \"3647c8cf-e471-42cf-ae45-02eea81f7c06\") " pod="opendatahub/test-curl-9057-b2a70d"
Sep 17 11:08:43.133347 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:43.133323    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6j8x\" (UniqueName: \"kubernetes.io/projected/3647c8cf-e471-42cf-ae45-02eea81f7c06-kube-api-access-p6j8x\") pod \"test-curl-9057-b2a70d\" (UID: \"3647c8cf-e471-42cf-ae45-02eea81f7c06\") " pod="opendatahub/test-curl-9057-b2a70d"
Sep 17 11:08:43.255179 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:43.255084    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9057-b2a70d"
Sep 17 11:08:43.533223 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:43.533200    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9057-b2a70d"]
Sep 17 11:08:43.535609 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:08:43.535585    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3647c8cf_e471_42cf_ae45_02eea81f7c06.slice/crio-59da00574a78945e64933e4ae313860ee90ea62dd4c56a223d79fb7f49a16d30 WatchSource:0}: Error finding container 59da00574a78945e64933e4ae313860ee90ea62dd4c56a223d79fb7f49a16d30: Status 404 returned error can't find the container with id 59da00574a78945e64933e4ae313860ee90ea62dd4c56a223d79fb7f49a16d30
Sep 17 11:08:44.123302 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:44.123262    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9057-b2a70d" event={"ID":"3647c8cf-e471-42cf-ae45-02eea81f7c06","Type":"ContainerStarted","Data":"a10758acf39ffce0db496d140916ea4ce598a27627ee8e36e43cb6de8dc8a639"}
Sep 17 11:08:44.123757 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:44.123313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9057-b2a70d" event={"ID":"3647c8cf-e471-42cf-ae45-02eea81f7c06","Type":"ContainerStarted","Data":"59da00574a78945e64933e4ae313860ee90ea62dd4c56a223d79fb7f49a16d30"}
Sep 17 11:08:44.123757 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:44.123357    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9057-baa383" podUID="923d5e8e-2b94-44bb-9aca-c68d43e4e468" containerName="test-curl-9057-baa383" containerID="cri-o://1d507faa1bc0a1f7011979b1dacb4c8f17b25b01bca103366b83394ac9832aa8" gracePeriod=2
Sep 17 11:08:44.139456 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:44.139399    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9057-b2a70d" podStartSLOduration=2.139380778 podStartE2EDuration="2.139380778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:08:44.138891276 +0000 UTC m=+918.236936841" watchObservedRunningTime="2026-09-17 11:08:44.139380778 +0000 UTC m=+918.237426347"
Sep 17 11:08:44.141156 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:08:44.141128    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9057-baa383\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="923d5e8e-2b94-44bb-9aca-c68d43e4e468" pod="opendatahub/test-curl-9057-baa383"
Sep 17 11:08:44.143552 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:08:44.143532    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9057-baa383\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="923d5e8e-2b94-44bb-9aca-c68d43e4e468" pod="opendatahub/test-curl-9057-baa383"
Sep 17 11:08:44.846632 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:44.846598    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9057-b2a70d"]
Sep 17 11:08:44.850237 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:44.850210    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9057-b2a70d"]
Sep 17 11:08:44.891541 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:44.891457    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:08:45.128649 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:08:45.128619    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9057-b2a70d" container="test-curl-9057-b2a70d"
Sep 17 11:08:46.134195 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:46.134159    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9057-b2a70d" podUID="3647c8cf-e471-42cf-ae45-02eea81f7c06" containerName="test-curl-9057-b2a70d" containerID="cri-o://a10758acf39ffce0db496d140916ea4ce598a27627ee8e36e43cb6de8dc8a639" gracePeriod=2
Sep 17 11:08:46.136895 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:08:46.136842    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9057-b2a70d\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="3647c8cf-e471-42cf-ae45-02eea81f7c06" pod="opendatahub/test-curl-9057-b2a70d"
Sep 17 11:08:46.301046 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:46.301024    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9057-baa383"
Sep 17 11:08:46.303708 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:08:46.303682    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9057-b2a70d\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="3647c8cf-e471-42cf-ae45-02eea81f7c06" pod="opendatahub/test-curl-9057-b2a70d"
Sep 17 11:08:46.349206 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:46.349176    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlg4d\" (UniqueName: \"kubernetes.io/projected/923d5e8e-2b94-44bb-9aca-c68d43e4e468-kube-api-access-vlg4d\") pod \"923d5e8e-2b94-44bb-9aca-c68d43e4e468\" (UID: \"923d5e8e-2b94-44bb-9aca-c68d43e4e468\") "
Sep 17 11:08:46.351381 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:46.351353    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/923d5e8e-2b94-44bb-9aca-c68d43e4e468-kube-api-access-vlg4d" (OuterVolumeSpecName: "kube-api-access-vlg4d") pod "923d5e8e-2b94-44bb-9aca-c68d43e4e468" (UID: "923d5e8e-2b94-44bb-9aca-c68d43e4e468"). InnerVolumeSpecName "kube-api-access-vlg4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:08:46.450764 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:46.450676    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlg4d\" (UniqueName: \"kubernetes.io/projected/923d5e8e-2b94-44bb-9aca-c68d43e4e468-kube-api-access-vlg4d\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:08:46.454373 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:08:46.454341    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9057-b2a70d\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="3647c8cf-e471-42cf-ae45-02eea81f7c06" pod="opendatahub/test-curl-9057-b2a70d"
Sep 17 11:08:46.456138 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:46.456113    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="923d5e8e-2b94-44bb-9aca-c68d43e4e468" path="/var/lib/kubelet/pods/923d5e8e-2b94-44bb-9aca-c68d43e4e468/volumes"
Sep 17 11:08:47.140266 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:47.140213    2562 generic.go:356] "Generic (PLEG): container finished" podID="923d5e8e-2b94-44bb-9aca-c68d43e4e468" containerID="1d507faa1bc0a1f7011979b1dacb4c8f17b25b01bca103366b83394ac9832aa8" exitCode=137
Sep 17 11:08:47.140266 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:47.140260    2562 scope.go:117] "RemoveContainer" containerID="1d507faa1bc0a1f7011979b1dacb4c8f17b25b01bca103366b83394ac9832aa8"
Sep 17 11:08:47.140266 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:47.140269    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9057-baa383"
Sep 17 11:08:47.149055 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:47.149036    2562 scope.go:117] "RemoveContainer" containerID="1d507faa1bc0a1f7011979b1dacb4c8f17b25b01bca103366b83394ac9832aa8"
Sep 17 11:08:47.149335 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:08:47.149314    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d507faa1bc0a1f7011979b1dacb4c8f17b25b01bca103366b83394ac9832aa8\": container with ID starting with 1d507faa1bc0a1f7011979b1dacb4c8f17b25b01bca103366b83394ac9832aa8 not found: ID does not exist" containerID="1d507faa1bc0a1f7011979b1dacb4c8f17b25b01bca103366b83394ac9832aa8"
Sep 17 11:08:47.149384 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:47.149345    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d507faa1bc0a1f7011979b1dacb4c8f17b25b01bca103366b83394ac9832aa8"} err="failed to get container status \"1d507faa1bc0a1f7011979b1dacb4c8f17b25b01bca103366b83394ac9832aa8\": rpc error: code = NotFound desc = could not find container \"1d507faa1bc0a1f7011979b1dacb4c8f17b25b01bca103366b83394ac9832aa8\": container with ID starting with 1d507faa1bc0a1f7011979b1dacb4c8f17b25b01bca103366b83394ac9832aa8 not found: ID does not exist"
Sep 17 11:08:47.741129 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:47.741098    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:08:48.311938 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:48.311912    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9057-b2a70d"
Sep 17 11:08:48.371625 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:48.371587    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6j8x\" (UniqueName: \"kubernetes.io/projected/3647c8cf-e471-42cf-ae45-02eea81f7c06-kube-api-access-p6j8x\") pod \"3647c8cf-e471-42cf-ae45-02eea81f7c06\" (UID: \"3647c8cf-e471-42cf-ae45-02eea81f7c06\") "
Sep 17 11:08:48.373622 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:48.373595    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3647c8cf-e471-42cf-ae45-02eea81f7c06-kube-api-access-p6j8x" (OuterVolumeSpecName: "kube-api-access-p6j8x") pod "3647c8cf-e471-42cf-ae45-02eea81f7c06" (UID: "3647c8cf-e471-42cf-ae45-02eea81f7c06"). InnerVolumeSpecName "kube-api-access-p6j8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:08:48.456476 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:48.456395    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3647c8cf-e471-42cf-ae45-02eea81f7c06" path="/var/lib/kubelet/pods/3647c8cf-e471-42cf-ae45-02eea81f7c06/volumes"
Sep 17 11:08:48.473271 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:48.473243    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6j8x\" (UniqueName: \"kubernetes.io/projected/3647c8cf-e471-42cf-ae45-02eea81f7c06-kube-api-access-p6j8x\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:08:49.152584 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:49.152543    2562 generic.go:356] "Generic (PLEG): container finished" podID="3647c8cf-e471-42cf-ae45-02eea81f7c06" containerID="a10758acf39ffce0db496d140916ea4ce598a27627ee8e36e43cb6de8dc8a639" exitCode=137
Sep 17 11:08:49.152584 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:49.152583    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9057-b2a70d"
Sep 17 11:08:49.152809 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:49.152612    2562 scope.go:117] "RemoveContainer" containerID="a10758acf39ffce0db496d140916ea4ce598a27627ee8e36e43cb6de8dc8a639"
Sep 17 11:08:49.161174 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:49.161157    2562 scope.go:117] "RemoveContainer" containerID="a10758acf39ffce0db496d140916ea4ce598a27627ee8e36e43cb6de8dc8a639"
Sep 17 11:08:49.161433 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:08:49.161417    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a10758acf39ffce0db496d140916ea4ce598a27627ee8e36e43cb6de8dc8a639\": container with ID starting with a10758acf39ffce0db496d140916ea4ce598a27627ee8e36e43cb6de8dc8a639 not found: ID does not exist" containerID="a10758acf39ffce0db496d140916ea4ce598a27627ee8e36e43cb6de8dc8a639"
Sep 17 11:08:49.161477 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:49.161439    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a10758acf39ffce0db496d140916ea4ce598a27627ee8e36e43cb6de8dc8a639"} err="failed to get container status \"a10758acf39ffce0db496d140916ea4ce598a27627ee8e36e43cb6de8dc8a639\": rpc error: code = NotFound desc = could not find container \"a10758acf39ffce0db496d140916ea4ce598a27627ee8e36e43cb6de8dc8a639\": container with ID starting with a10758acf39ffce0db496d140916ea4ce598a27627ee8e36e43cb6de8dc8a639 not found: ID does not exist"
Sep 17 11:08:49.830431 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:08:49.830391    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:09:09.198151 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.198115    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"]
Sep 17 11:09:09.198556 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.198528    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3647c8cf-e471-42cf-ae45-02eea81f7c06" containerName="test-curl-9057-b2a70d"
Sep 17 11:09:09.198556 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.198540    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3647c8cf-e471-42cf-ae45-02eea81f7c06" containerName="test-curl-9057-b2a70d"
Sep 17 11:09:09.198634 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.198620    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3647c8cf-e471-42cf-ae45-02eea81f7c06" containerName="test-curl-9057-b2a70d"
Sep 17 11:09:09.202124 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.202107    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.223992 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.223965    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-766cc3\"/\"default-dockercfg-kcxrj\""
Sep 17 11:09:09.225079 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.225065    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-766cc3\"/\"distinct-w3-766cc3-kserve-self-signed-certs\""
Sep 17 11:09:09.233562 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.233546    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-766cc3\"/\"openshift-service-ca.crt\""
Sep 17 11:09:09.238537 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.238516    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-766cc3\"/\"kube-root-ca.crt\""
Sep 17 11:09:09.260475 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.260444    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"]
Sep 17 11:09:09.335587 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.335549    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:09.372279 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.372243    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-tmp-dir\") pod \"distinct-w3-766cc3-kserve-7777dd6596-7ss54\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") " pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.372457 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.372289    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-home\") pod \"distinct-w3-766cc3-kserve-7777dd6596-7ss54\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") " pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.372457 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.372402    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-dshm\") pod \"distinct-w3-766cc3-kserve-7777dd6596-7ss54\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") " pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.372457 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.372436    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f19bbdec-e2e1-485a-aab3-c9fda81702a8-tls-certs\") pod \"distinct-w3-766cc3-kserve-7777dd6596-7ss54\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") " pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.372591 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.372467    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph7s8\" (UniqueName: \"kubernetes.io/projected/f19bbdec-e2e1-485a-aab3-c9fda81702a8-kube-api-access-ph7s8\") pod \"distinct-w3-766cc3-kserve-7777dd6596-7ss54\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") " pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.372591 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.372547    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-model-cache\") pod \"distinct-w3-766cc3-kserve-7777dd6596-7ss54\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") " pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.473379 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.473292    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-dshm\") pod \"distinct-w3-766cc3-kserve-7777dd6596-7ss54\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") " pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.473379 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.473334    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f19bbdec-e2e1-485a-aab3-c9fda81702a8-tls-certs\") pod \"distinct-w3-766cc3-kserve-7777dd6596-7ss54\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") " pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.473379 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.473370    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph7s8\" (UniqueName: \"kubernetes.io/projected/f19bbdec-e2e1-485a-aab3-c9fda81702a8-kube-api-access-ph7s8\") pod \"distinct-w3-766cc3-kserve-7777dd6596-7ss54\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") " pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.473631 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.473422    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-model-cache\") pod \"distinct-w3-766cc3-kserve-7777dd6596-7ss54\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") " pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.473631 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.473485    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-tmp-dir\") pod \"distinct-w3-766cc3-kserve-7777dd6596-7ss54\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") " pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.473631 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.473514    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-home\") pod \"distinct-w3-766cc3-kserve-7777dd6596-7ss54\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") " pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.473875 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.473846    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-model-cache\") pod \"distinct-w3-766cc3-kserve-7777dd6596-7ss54\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") " pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.473950 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.473889    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-tmp-dir\") pod \"distinct-w3-766cc3-kserve-7777dd6596-7ss54\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") " pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.473994 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.473945    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-home\") pod \"distinct-w3-766cc3-kserve-7777dd6596-7ss54\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") " pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.475592 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.475574    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-dshm\") pod \"distinct-w3-766cc3-kserve-7777dd6596-7ss54\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") " pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.475778 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.475759    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f19bbdec-e2e1-485a-aab3-c9fda81702a8-tls-certs\") pod \"distinct-w3-766cc3-kserve-7777dd6596-7ss54\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") " pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.485695 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.485671    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph7s8\" (UniqueName: \"kubernetes.io/projected/f19bbdec-e2e1-485a-aab3-c9fda81702a8-kube-api-access-ph7s8\") pod \"distinct-w3-766cc3-kserve-7777dd6596-7ss54\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") " pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.511584 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.511553    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:09.825519 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:09.825210    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"]
Sep 17 11:09:09.826511 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:09:09.826463    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf19bbdec_e2e1_485a_aab3_c9fda81702a8.slice/crio-a3f98e4ff1845a6f47233c381f98fe5a0c3cf768954bec4ef852231d6c4f761d WatchSource:0}: Error finding container a3f98e4ff1845a6f47233c381f98fe5a0c3cf768954bec4ef852231d6c4f761d: Status 404 returned error can't find the container with id a3f98e4ff1845a6f47233c381f98fe5a0c3cf768954bec4ef852231d6c4f761d
Sep 17 11:09:10.271405 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:10.271321    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54" event={"ID":"f19bbdec-e2e1-485a-aab3-c9fda81702a8","Type":"ContainerStarted","Data":"8aaea066bb961edcf8803012bc4ba3960b0c5915aa656f4b53348f045b151377"}
Sep 17 11:09:10.271405 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:10.271361    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54" event={"ID":"f19bbdec-e2e1-485a-aab3-c9fda81702a8","Type":"ContainerStarted","Data":"a3f98e4ff1845a6f47233c381f98fe5a0c3cf768954bec4ef852231d6c4f761d"}
Sep 17 11:09:10.304934 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:10.304884    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54" podStartSLOduration=1.304868172 podStartE2EDuration="1.304868172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:09:10.300506988 +0000 UTC m=+944.398552553" watchObservedRunningTime="2026-09-17 11:09:10.304868172 +0000 UTC m=+944.402913738"
Sep 17 11:09:13.579113 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:13.579077    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:09:18.787332 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:18.787262    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:19.511698 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:19.511658    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:19.511908 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:19.511741    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:19.524283 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:19.524255    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:19.711331 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:19.711292    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:20.335578 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:20.335545    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:09:29.198259 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:29.198214    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:29.324075 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:29.324037    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:33.306311 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:33.306265    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:33.433463 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:33.433423    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:33.807679 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:33.807626    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:39.127529 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:39.127483    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:09:39.832659 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:39.832621    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:39.946978 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:39.946943    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:40.177796 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:40.177726    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:40.524209 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:40.524128    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:40.671880 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:40.671844    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:42.982935 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:42.982900    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:43.660440 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:43.660408    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:43.779282 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:43.779242    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:44.249912 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:44.249879    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:45.082800 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:45.082764    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:46.202073 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:46.202037    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:46.964525 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:46.964488    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"]
Sep 17 11:09:46.970139 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:46.970115    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:09:46.973148 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:46.973124    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-dfe3d913-serving-cert\""
Sep 17 11:09:46.980715 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:46.980689    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"]
Sep 17 11:09:47.000160 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.000122    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:47.141255 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.141218    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:47.146436 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.146416    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkxhp\" (UniqueName: \"kubernetes.io/projected/c0407e8a-1dd8-474c-a721-bdfee729bbcd-kube-api-access-fkxhp\") pod \"maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb\" (UID: \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:09:47.146511 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.146453    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0407e8a-1dd8-474c-a721-bdfee729bbcd-openshift-service-ca\") pod \"maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb\" (UID: \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:09:47.146511 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.146473    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0407e8a-1dd8-474c-a721-bdfee729bbcd-maas-api-tls\") pod \"maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb\" (UID: \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:09:47.146591 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.146545    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0407e8a-1dd8-474c-a721-bdfee729bbcd-metrics-certs\") pod \"maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb\" (UID: \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:09:47.248131 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.248045    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0407e8a-1dd8-474c-a721-bdfee729bbcd-openshift-service-ca\") pod \"maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb\" (UID: \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:09:47.248131 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.248082    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0407e8a-1dd8-474c-a721-bdfee729bbcd-maas-api-tls\") pod \"maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb\" (UID: \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:09:47.248537 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.248162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0407e8a-1dd8-474c-a721-bdfee729bbcd-metrics-certs\") pod \"maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb\" (UID: \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:09:47.248537 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.248259    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkxhp\" (UniqueName: \"kubernetes.io/projected/c0407e8a-1dd8-474c-a721-bdfee729bbcd-kube-api-access-fkxhp\") pod \"maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb\" (UID: \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:09:47.248537 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:09:47.248269    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-serving-cert: secret "maas-api-e2e-ait-dfe3d913-serving-cert" not found
Sep 17 11:09:47.248537 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:09:47.248338    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0407e8a-1dd8-474c-a721-bdfee729bbcd-maas-api-tls podName:c0407e8a-1dd8-474c-a721-bdfee729bbcd nodeName:}" failed. No retries permitted until 2026-09-17 11:09:47.748318003 +0000 UTC m=+981.846363547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0407e8a-1dd8-474c-a721-bdfee729bbcd-maas-api-tls") pod "maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb" (UID: "c0407e8a-1dd8-474c-a721-bdfee729bbcd") : secret "maas-api-e2e-ait-dfe3d913-serving-cert" not found
Sep 17 11:09:47.248810 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.248792    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0407e8a-1dd8-474c-a721-bdfee729bbcd-openshift-service-ca\") pod \"maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb\" (UID: \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:09:47.250541 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.250521    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0407e8a-1dd8-474c-a721-bdfee729bbcd-metrics-certs\") pod \"maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb\" (UID: \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:09:47.271890 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.271856    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkxhp\" (UniqueName: \"kubernetes.io/projected/c0407e8a-1dd8-474c-a721-bdfee729bbcd-kube-api-access-fkxhp\") pod \"maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb\" (UID: \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:09:47.501709 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.501613    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:47.750938 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.750911    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:47.753258 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.753185    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0407e8a-1dd8-474c-a721-bdfee729bbcd-maas-api-tls\") pod \"maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb\" (UID: \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:09:47.756002 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.755979    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0407e8a-1dd8-474c-a721-bdfee729bbcd-maas-api-tls\") pod \"maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb\" (UID: \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:09:47.843299 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.843266    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:47.881676 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:47.881639    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:09:48.160067 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:48.160043    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"]
Sep 17 11:09:48.161918 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:09:48.161896    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0407e8a_1dd8_474c_a721_bdfee729bbcd.slice/crio-de2813f3ddc804c4c163e4c9d01e9a73f22f373313ecf75faf5465497b953dd3 WatchSource:0}: Error finding container de2813f3ddc804c4c163e4c9d01e9a73f22f373313ecf75faf5465497b953dd3: Status 404 returned error can't find the container with id de2813f3ddc804c4c163e4c9d01e9a73f22f373313ecf75faf5465497b953dd3
Sep 17 11:09:48.481314 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:48.481247    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb" event={"ID":"c0407e8a-1dd8-474c-a721-bdfee729bbcd","Type":"ContainerStarted","Data":"de2813f3ddc804c4c163e4c9d01e9a73f22f373313ecf75faf5465497b953dd3"}
Sep 17 11:09:49.487725 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:49.487629    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb" event={"ID":"c0407e8a-1dd8-474c-a721-bdfee729bbcd","Type":"ContainerStarted","Data":"a46dc1f9a60d1ffab01aed15325c1a65c593870452fde5a6868be367c31152e2"}
Sep 17 11:09:49.487725 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:49.487720    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:09:49.509371 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:49.509324    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb" podStartSLOduration=3.173795343 podStartE2EDuration="3.509308301s" totalImagesPullingTime="335.512958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:09:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:09:49.508337824 +0000 UTC m=+983.606383389" watchObservedRunningTime="2026-09-17 11:09:49.509308301 +0000 UTC m=+983.607353867"
Sep 17 11:09:49.846167 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:49.846131    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:49.910022 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:49.909991    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:50.499941 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:50.499911    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:09:55.498343 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:09:55.498310    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:10:00.715741 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:00.715700    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:10:09.789389 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:09.789344    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:10.641125 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:10.641088    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:11.461345 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:11.461311    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:12.219359 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:12.219318    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:13.060330 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:13.060298    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:15.345988 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:15.345951    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:16.068991 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:16.068958    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:16.537033 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:16.536999    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:17.128564 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:17.128529    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:17.982898 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:17.982863    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:18.755117 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:18.755082    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:25.523016 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.522987    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"]
Sep 17 11:10:25.523411 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.523190    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb" podUID="c0407e8a-1dd8-474c-a721-bdfee729bbcd" containerName="maas-api" containerID="cri-o://a46dc1f9a60d1ffab01aed15325c1a65c593870452fde5a6868be367c31152e2" gracePeriod=30
Sep 17 11:10:25.623172 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.623128    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:25.678259 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.678213    2562 generic.go:356] "Generic (PLEG): container finished" podID="c0407e8a-1dd8-474c-a721-bdfee729bbcd" containerID="a46dc1f9a60d1ffab01aed15325c1a65c593870452fde5a6868be367c31152e2" exitCode=0
Sep 17 11:10:25.678434 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.678307    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb" event={"ID":"c0407e8a-1dd8-474c-a721-bdfee729bbcd","Type":"ContainerDied","Data":"a46dc1f9a60d1ffab01aed15325c1a65c593870452fde5a6868be367c31152e2"}
Sep 17 11:10:25.770983 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.770960    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:25.787336 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.787310    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:10:25.833949 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.833920    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkxhp\" (UniqueName: \"kubernetes.io/projected/c0407e8a-1dd8-474c-a721-bdfee729bbcd-kube-api-access-fkxhp\") pod \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\" (UID: \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\") "
Sep 17 11:10:25.834113 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.833973    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0407e8a-1dd8-474c-a721-bdfee729bbcd-maas-api-tls\") pod \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\" (UID: \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\") "
Sep 17 11:10:25.834113 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.834067    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0407e8a-1dd8-474c-a721-bdfee729bbcd-openshift-service-ca\") pod \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\" (UID: \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\") "
Sep 17 11:10:25.834230 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.834131    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0407e8a-1dd8-474c-a721-bdfee729bbcd-metrics-certs\") pod \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\" (UID: \"c0407e8a-1dd8-474c-a721-bdfee729bbcd\") "
Sep 17 11:10:25.834472 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.834439    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0407e8a-1dd8-474c-a721-bdfee729bbcd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c0407e8a-1dd8-474c-a721-bdfee729bbcd" (UID: "c0407e8a-1dd8-474c-a721-bdfee729bbcd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:10:25.836252 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.836222    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0407e8a-1dd8-474c-a721-bdfee729bbcd-kube-api-access-fkxhp" (OuterVolumeSpecName: "kube-api-access-fkxhp") pod "c0407e8a-1dd8-474c-a721-bdfee729bbcd" (UID: "c0407e8a-1dd8-474c-a721-bdfee729bbcd"). InnerVolumeSpecName "kube-api-access-fkxhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:10:25.836354 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.836268    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0407e8a-1dd8-474c-a721-bdfee729bbcd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c0407e8a-1dd8-474c-a721-bdfee729bbcd" (UID: "c0407e8a-1dd8-474c-a721-bdfee729bbcd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:10:25.836354 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.836308    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0407e8a-1dd8-474c-a721-bdfee729bbcd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c0407e8a-1dd8-474c-a721-bdfee729bbcd" (UID: "c0407e8a-1dd8-474c-a721-bdfee729bbcd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:10:25.921300 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.921260    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:25.935851 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.935803    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkxhp\" (UniqueName: \"kubernetes.io/projected/c0407e8a-1dd8-474c-a721-bdfee729bbcd-kube-api-access-fkxhp\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:10:25.935851 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.935851    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0407e8a-1dd8-474c-a721-bdfee729bbcd-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:10:25.935980 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.935861    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0407e8a-1dd8-474c-a721-bdfee729bbcd-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:10:25.935980 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:25.935870    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0407e8a-1dd8-474c-a721-bdfee729bbcd-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:10:26.079393 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:26.079307    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:26.181869 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:26.181828    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:26.301113 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:26.301085    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:26.685079 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:26.685042    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"
Sep 17 11:10:26.685521 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:26.685078    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb" event={"ID":"c0407e8a-1dd8-474c-a721-bdfee729bbcd","Type":"ContainerDied","Data":"de2813f3ddc804c4c163e4c9d01e9a73f22f373313ecf75faf5465497b953dd3"}
Sep 17 11:10:26.685521 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:26.685153    2562 scope.go:117] "RemoveContainer" containerID="a46dc1f9a60d1ffab01aed15325c1a65c593870452fde5a6868be367c31152e2"
Sep 17 11:10:26.721404 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:26.721320    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"]
Sep 17 11:10:26.727549 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:26.727524    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dfe3d913-5f9d9cdf5c-wjnnb"]
Sep 17 11:10:26.775237 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:26.775205    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:26.876794 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:26.876757    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:26.983555 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:26.983469    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:27.083035 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:27.083004    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:27.380156 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:27.380122    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:27.485291 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:27.485254    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:27.975285 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:27.975256    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:28.457098 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:28.457069    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0407e8a-1dd8-474c-a721-bdfee729bbcd" path="/var/lib/kubelet/pods/c0407e8a-1dd8-474c-a721-bdfee729bbcd/volumes"
Sep 17 11:10:28.781595 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:28.781567    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:29.719479 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:29.719446    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:30.069353 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.069319    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:30.169932 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.169904    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:30.778729 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.778698    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"]
Sep 17 11:10:30.779168 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.779152    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0407e8a-1dd8-474c-a721-bdfee729bbcd" containerName="maas-api"
Sep 17 11:10:30.779168 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.779169    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0407e8a-1dd8-474c-a721-bdfee729bbcd" containerName="maas-api"
Sep 17 11:10:30.779281 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.779250    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0407e8a-1dd8-474c-a721-bdfee729bbcd" containerName="maas-api"
Sep 17 11:10:30.783676 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.783652    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"
Sep 17 11:10:30.786881 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.786859    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-76b12dde-serving-cert\""
Sep 17 11:10:30.830696 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.830658    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"]
Sep 17 11:10:30.882285 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.882244    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-openshift-service-ca\") pod \"maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2\" (UID: \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"
Sep 17 11:10:30.882477 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.882298    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr59m\" (UniqueName: \"kubernetes.io/projected/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-kube-api-access-wr59m\") pod \"maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2\" (UID: \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"
Sep 17 11:10:30.882477 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.882364    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-maas-api-tls\") pod \"maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2\" (UID: \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"
Sep 17 11:10:30.882477 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.882382    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-metrics-certs\") pod \"maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2\" (UID: \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"
Sep 17 11:10:30.969760 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.969727    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:30.982985 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.982957    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-openshift-service-ca\") pod \"maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2\" (UID: \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"
Sep 17 11:10:30.983064 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.983020    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr59m\" (UniqueName: \"kubernetes.io/projected/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-kube-api-access-wr59m\") pod \"maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2\" (UID: \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"
Sep 17 11:10:30.983119 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.983066    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-maas-api-tls\") pod \"maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2\" (UID: \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"
Sep 17 11:10:30.983119 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.983084    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-metrics-certs\") pod \"maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2\" (UID: \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"
Sep 17 11:10:30.983696 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.983672    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-openshift-service-ca\") pod \"maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2\" (UID: \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"
Sep 17 11:10:30.985538 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.985511    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-metrics-certs\") pod \"maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2\" (UID: \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"
Sep 17 11:10:30.985637 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:30.985541    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-maas-api-tls\") pod \"maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2\" (UID: \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"
Sep 17 11:10:31.013212 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:31.013177    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr59m\" (UniqueName: \"kubernetes.io/projected/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-kube-api-access-wr59m\") pod \"maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2\" (UID: \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"
Sep 17 11:10:31.095928 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:31.095839    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"
Sep 17 11:10:31.184940 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:31.184893    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:31.368609 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:31.368584    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"]
Sep 17 11:10:31.370595 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:10:31.370561    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f4d3af_4ba7_4b5d_9e05_ec5dba3d5ce7.slice/crio-2d4831cfcfc79da98cb634b432b25e3af2d6411ef5c1f20899312a0724b370dd WatchSource:0}: Error finding container 2d4831cfcfc79da98cb634b432b25e3af2d6411ef5c1f20899312a0724b370dd: Status 404 returned error can't find the container with id 2d4831cfcfc79da98cb634b432b25e3af2d6411ef5c1f20899312a0724b370dd
Sep 17 11:10:31.680658 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:31.680537    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:31.713547 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:31.713509    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2" event={"ID":"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7","Type":"ContainerStarted","Data":"2d4831cfcfc79da98cb634b432b25e3af2d6411ef5c1f20899312a0724b370dd"}
Sep 17 11:10:31.771525 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:31.771496    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:32.362874 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:32.362803    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:32.723423 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:32.723324    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2" event={"ID":"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7","Type":"ContainerStarted","Data":"fd420734359330842936bdda401a4f80ca23ebb8442bd214c811640a9d4e0ecf"}
Sep 17 11:10:32.723585 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:32.723510    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"
Sep 17 11:10:32.746781 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:32.746731    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2" podStartSLOduration=2.17307796 podStartE2EDuration="2.746715527s" totalImagesPullingTime="573.637567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:10:32.745840223 +0000 UTC m=+1026.843885791" watchObservedRunningTime="2026-09-17 11:10:32.746715527 +0000 UTC m=+1026.844761092"
Sep 17 11:10:33.212628 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:33.212595    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:33.943591 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:33.943563    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:34.866109 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:34.866075    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:35.617940 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:35.617905    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:36.019447 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:36.019359    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:10:38.734041 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:38.734005    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"
Sep 17 11:10:49.821071 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:49.821034    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"]
Sep 17 11:10:49.854882 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:49.854846    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:53.648405 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:53.648362    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:54.463608 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:54.463568    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:55.550305 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:55.550274    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:56.339013 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:56.338966    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:57.129784 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:57.129739    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:58.026671 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:58.026641    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:58.993590 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:58.993560    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:10:59.842912 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:10:59.842871    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:01.130825 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:01.130795    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:01.893081 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:01.893039    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"]
Sep 17 11:11:01.893303 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:01.893274    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2" podUID="a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7" containerName="maas-api" containerID="cri-o://fd420734359330842936bdda401a4f80ca23ebb8442bd214c811640a9d4e0ecf" gracePeriod=30
Sep 17 11:11:01.929971 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:01.929938    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:02.014387 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.014356    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:02.122967 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.122937    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:02.159914 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.159846    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"
Sep 17 11:11:02.185291 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.185254    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-metrics-certs\") pod \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\" (UID: \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\") "
Sep 17 11:11:02.185527 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.185302    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-openshift-service-ca\") pod \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\" (UID: \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\") "
Sep 17 11:11:02.185527 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.185338    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr59m\" (UniqueName: \"kubernetes.io/projected/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-kube-api-access-wr59m\") pod \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\" (UID: \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\") "
Sep 17 11:11:02.185527 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.185457    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-maas-api-tls\") pod \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\" (UID: \"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7\") "
Sep 17 11:11:02.186206 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.186153    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7" (UID: "a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:11:02.188679 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.188279    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7" (UID: "a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:11:02.188679 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.188463    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-kube-api-access-wr59m" (OuterVolumeSpecName: "kube-api-access-wr59m") pod "a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7" (UID: "a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7"). InnerVolumeSpecName "kube-api-access-wr59m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:11:02.188679 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.188627    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7" (UID: "a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:11:02.215428 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.215390    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:02.287098 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.287053    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:02.287098 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.287093    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:02.287098 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.287107    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr59m\" (UniqueName: \"kubernetes.io/projected/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-kube-api-access-wr59m\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:02.287338 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.287121    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:02.358136 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.358099    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:02.867993 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.867964    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:02.886734 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.886698    2562 generic.go:356] "Generic (PLEG): container finished" podID="a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7" containerID="fd420734359330842936bdda401a4f80ca23ebb8442bd214c811640a9d4e0ecf" exitCode=0
Sep 17 11:11:02.886938 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.886741    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"
Sep 17 11:11:02.886938 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.886764    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2" event={"ID":"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7","Type":"ContainerDied","Data":"fd420734359330842936bdda401a4f80ca23ebb8442bd214c811640a9d4e0ecf"}
Sep 17 11:11:02.886938 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.886811    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2" event={"ID":"a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7","Type":"ContainerDied","Data":"2d4831cfcfc79da98cb634b432b25e3af2d6411ef5c1f20899312a0724b370dd"}
Sep 17 11:11:02.886938 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.886845    2562 scope.go:117] "RemoveContainer" containerID="fd420734359330842936bdda401a4f80ca23ebb8442bd214c811640a9d4e0ecf"
Sep 17 11:11:02.896247 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.896067    2562 scope.go:117] "RemoveContainer" containerID="fd420734359330842936bdda401a4f80ca23ebb8442bd214c811640a9d4e0ecf"
Sep 17 11:11:02.896395 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:11:02.896373    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd420734359330842936bdda401a4f80ca23ebb8442bd214c811640a9d4e0ecf\": container with ID starting with fd420734359330842936bdda401a4f80ca23ebb8442bd214c811640a9d4e0ecf not found: ID does not exist" containerID="fd420734359330842936bdda401a4f80ca23ebb8442bd214c811640a9d4e0ecf"
Sep 17 11:11:02.896442 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.896404    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd420734359330842936bdda401a4f80ca23ebb8442bd214c811640a9d4e0ecf"} err="failed to get container status \"fd420734359330842936bdda401a4f80ca23ebb8442bd214c811640a9d4e0ecf\": rpc error: code = NotFound desc = could not find container \"fd420734359330842936bdda401a4f80ca23ebb8442bd214c811640a9d4e0ecf\": container with ID starting with fd420734359330842936bdda401a4f80ca23ebb8442bd214c811640a9d4e0ecf not found: ID does not exist"
Sep 17 11:11:02.909512 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.909473    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"]
Sep 17 11:11:02.915600 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:02.915572    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-76b12dde-6bbb7cf596-wk8x2"]
Sep 17 11:11:03.173866 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:03.173745    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:03.446852 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:03.446740    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:03.548287 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:03.548250    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:04.279009 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:04.278973    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:04.456385 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:04.456310    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7" path="/var/lib/kubelet/pods/a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7/volumes"
Sep 17 11:11:04.840197 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:04.840153    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88" podUID="53c60f88-cc3f-4e03-a6cd-4913652be060" containerName="main" containerID="cri-o://4d35f80aff1a2f642ed49f50a4697263ea63804b19abe36bb09e1e512d0106de" gracePeriod=45
Sep 17 11:11:05.103679 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.103608    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:11:05.218525 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.218486    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-dshm\") pod \"53c60f88-cc3f-4e03-a6cd-4913652be060\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") "
Sep 17 11:11:05.218730 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.218585    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-tmp-dir\") pod \"53c60f88-cc3f-4e03-a6cd-4913652be060\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") "
Sep 17 11:11:05.218730 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.218619    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53c60f88-cc3f-4e03-a6cd-4913652be060-tls-certs\") pod \"53c60f88-cc3f-4e03-a6cd-4913652be060\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") "
Sep 17 11:11:05.218730 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.218657    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnjwh\" (UniqueName: \"kubernetes.io/projected/53c60f88-cc3f-4e03-a6cd-4913652be060-kube-api-access-wnjwh\") pod \"53c60f88-cc3f-4e03-a6cd-4913652be060\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") "
Sep 17 11:11:05.218730 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.218688    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-model-cache\") pod \"53c60f88-cc3f-4e03-a6cd-4913652be060\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") "
Sep 17 11:11:05.218955 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.218735    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-home\") pod \"53c60f88-cc3f-4e03-a6cd-4913652be060\" (UID: \"53c60f88-cc3f-4e03-a6cd-4913652be060\") "
Sep 17 11:11:05.218955 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.218864    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53c60f88-cc3f-4e03-a6cd-4913652be060" (UID: "53c60f88-cc3f-4e03-a6cd-4913652be060"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:11:05.219072 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.219027    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-model-cache" (OuterVolumeSpecName: "model-cache") pod "53c60f88-cc3f-4e03-a6cd-4913652be060" (UID: "53c60f88-cc3f-4e03-a6cd-4913652be060"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:11:05.219072 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.219044    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-home" (OuterVolumeSpecName: "home") pod "53c60f88-cc3f-4e03-a6cd-4913652be060" (UID: "53c60f88-cc3f-4e03-a6cd-4913652be060"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:11:05.219222 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.219196    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:05.219222 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.219225    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:05.219405 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.219239    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:05.220926 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.220901    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-dshm" (OuterVolumeSpecName: "dshm") pod "53c60f88-cc3f-4e03-a6cd-4913652be060" (UID: "53c60f88-cc3f-4e03-a6cd-4913652be060"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:11:05.221020 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.220938    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53c60f88-cc3f-4e03-a6cd-4913652be060-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53c60f88-cc3f-4e03-a6cd-4913652be060" (UID: "53c60f88-cc3f-4e03-a6cd-4913652be060"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:11:05.221020 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.220962    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c60f88-cc3f-4e03-a6cd-4913652be060-kube-api-access-wnjwh" (OuterVolumeSpecName: "kube-api-access-wnjwh") pod "53c60f88-cc3f-4e03-a6cd-4913652be060" (UID: "53c60f88-cc3f-4e03-a6cd-4913652be060"). InnerVolumeSpecName "kube-api-access-wnjwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:11:05.320680 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.320643    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53c60f88-cc3f-4e03-a6cd-4913652be060-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:05.320680 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.320674    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53c60f88-cc3f-4e03-a6cd-4913652be060-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:05.320680 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.320684    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnjwh\" (UniqueName: \"kubernetes.io/projected/53c60f88-cc3f-4e03-a6cd-4913652be060-kube-api-access-wnjwh\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:05.368168 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.368081    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:05.904544 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.904505    2562 generic.go:356] "Generic (PLEG): container finished" podID="53c60f88-cc3f-4e03-a6cd-4913652be060" containerID="4d35f80aff1a2f642ed49f50a4697263ea63804b19abe36bb09e1e512d0106de" exitCode=0
Sep 17 11:11:05.904719 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.904553    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"
Sep 17 11:11:05.904719 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.904591    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88" event={"ID":"53c60f88-cc3f-4e03-a6cd-4913652be060","Type":"ContainerDied","Data":"4d35f80aff1a2f642ed49f50a4697263ea63804b19abe36bb09e1e512d0106de"}
Sep 17 11:11:05.904719 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.904638    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88" event={"ID":"53c60f88-cc3f-4e03-a6cd-4913652be060","Type":"ContainerDied","Data":"d814ae357253d1ae69dcad5f4ae19895e517c970843a93151f1f71fda900e170"}
Sep 17 11:11:05.904719 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.904657    2562 scope.go:117] "RemoveContainer" containerID="4d35f80aff1a2f642ed49f50a4697263ea63804b19abe36bb09e1e512d0106de"
Sep 17 11:11:05.914326 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.914305    2562 scope.go:117] "RemoveContainer" containerID="4d35f80aff1a2f642ed49f50a4697263ea63804b19abe36bb09e1e512d0106de"
Sep 17 11:11:05.914600 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:11:05.914581    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d35f80aff1a2f642ed49f50a4697263ea63804b19abe36bb09e1e512d0106de\": container with ID starting with 4d35f80aff1a2f642ed49f50a4697263ea63804b19abe36bb09e1e512d0106de not found: ID does not exist" containerID="4d35f80aff1a2f642ed49f50a4697263ea63804b19abe36bb09e1e512d0106de"
Sep 17 11:11:05.914646 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.914608    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d35f80aff1a2f642ed49f50a4697263ea63804b19abe36bb09e1e512d0106de"} err="failed to get container status \"4d35f80aff1a2f642ed49f50a4697263ea63804b19abe36bb09e1e512d0106de\": rpc error: code = NotFound desc = could not find container \"4d35f80aff1a2f642ed49f50a4697263ea63804b19abe36bb09e1e512d0106de\": container with ID starting with 4d35f80aff1a2f642ed49f50a4697263ea63804b19abe36bb09e1e512d0106de not found: ID does not exist"
Sep 17 11:11:05.928446 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.928415    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"]
Sep 17 11:11:05.933058 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:05.933035    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-79a70a/test-model-79a70a-kserve-694494b495-5hh88"]
Sep 17 11:11:06.071136 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:06.071103    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:06.168320 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:06.168239    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:06.457062 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:06.456983    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53c60f88-cc3f-4e03-a6cd-4913652be060" path="/var/lib/kubelet/pods/53c60f88-cc3f-4e03-a6cd-4913652be060/volumes"
Sep 17 11:11:06.674559 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:06.674516    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:06.924967 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:06.924936    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:07.756259 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:07.756225    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:08.739833 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:08.739788    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:09.550714 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:09.550679    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:09.933358 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:09.933275    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:11:10.352481 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.352450    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"]
Sep 17 11:11:10.352895 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.352878    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c60f88-cc3f-4e03-a6cd-4913652be060" containerName="main"
Sep 17 11:11:10.352895 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.352895    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c60f88-cc3f-4e03-a6cd-4913652be060" containerName="main"
Sep 17 11:11:10.353007 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.352904    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7" containerName="maas-api"
Sep 17 11:11:10.353007 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.352910    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7" containerName="maas-api"
Sep 17 11:11:10.353007 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.352988    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c60f88-cc3f-4e03-a6cd-4913652be060" containerName="main"
Sep 17 11:11:10.353007 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.352999    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7f4d3af-4ba7-4b5d-9e05-ec5dba3d5ce7" containerName="maas-api"
Sep 17 11:11:10.356304 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.356280    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:10.360980 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.360958    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-15da354a-serving-cert\""
Sep 17 11:11:10.378267 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.378236    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"]
Sep 17 11:11:10.408522 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.408493    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:10.470082 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.470047    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljzg5\" (UniqueName: \"kubernetes.io/projected/66c2bf1c-95cc-43c0-9724-b89735374c99-kube-api-access-ljzg5\") pod \"maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h\" (UID: \"66c2bf1c-95cc-43c0-9724-b89735374c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:10.470241 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.470102    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66c2bf1c-95cc-43c0-9724-b89735374c99-maas-api-tls\") pod \"maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h\" (UID: \"66c2bf1c-95cc-43c0-9724-b89735374c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:10.470241 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.470123    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c2bf1c-95cc-43c0-9724-b89735374c99-openshift-service-ca\") pod \"maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h\" (UID: \"66c2bf1c-95cc-43c0-9724-b89735374c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:10.470241 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.470202    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66c2bf1c-95cc-43c0-9724-b89735374c99-metrics-certs\") pod \"maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h\" (UID: \"66c2bf1c-95cc-43c0-9724-b89735374c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:10.512282 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.512247    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:10.571287 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.571241    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66c2bf1c-95cc-43c0-9724-b89735374c99-metrics-certs\") pod \"maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h\" (UID: \"66c2bf1c-95cc-43c0-9724-b89735374c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:10.571763 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.571372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljzg5\" (UniqueName: \"kubernetes.io/projected/66c2bf1c-95cc-43c0-9724-b89735374c99-kube-api-access-ljzg5\") pod \"maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h\" (UID: \"66c2bf1c-95cc-43c0-9724-b89735374c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:10.571763 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.571442    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66c2bf1c-95cc-43c0-9724-b89735374c99-maas-api-tls\") pod \"maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h\" (UID: \"66c2bf1c-95cc-43c0-9724-b89735374c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:10.571763 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.571471    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c2bf1c-95cc-43c0-9724-b89735374c99-openshift-service-ca\") pod \"maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h\" (UID: \"66c2bf1c-95cc-43c0-9724-b89735374c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:10.571763 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:11:10.571555    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-serving-cert: secret "maas-api-e2e-ait-15da354a-serving-cert" not found
Sep 17 11:11:10.571763 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:11:10.571632    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66c2bf1c-95cc-43c0-9724-b89735374c99-maas-api-tls podName:66c2bf1c-95cc-43c0-9724-b89735374c99 nodeName:}" failed. No retries permitted until 2026-09-17 11:11:11.071610826 +0000 UTC m=+1065.169656391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66c2bf1c-95cc-43c0-9724-b89735374c99-maas-api-tls") pod "maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h" (UID: "66c2bf1c-95cc-43c0-9724-b89735374c99") : secret "maas-api-e2e-ait-15da354a-serving-cert" not found
Sep 17 11:11:10.572245 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.572224    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c2bf1c-95cc-43c0-9724-b89735374c99-openshift-service-ca\") pod \"maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h\" (UID: \"66c2bf1c-95cc-43c0-9724-b89735374c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:10.573723 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.573700    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66c2bf1c-95cc-43c0-9724-b89735374c99-metrics-certs\") pod \"maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h\" (UID: \"66c2bf1c-95cc-43c0-9724-b89735374c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:10.583515 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:10.583494    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljzg5\" (UniqueName: \"kubernetes.io/projected/66c2bf1c-95cc-43c0-9724-b89735374c99-kube-api-access-ljzg5\") pod \"maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h\" (UID: \"66c2bf1c-95cc-43c0-9724-b89735374c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:11.016787 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.016742    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:11.076938 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.076902    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66c2bf1c-95cc-43c0-9724-b89735374c99-maas-api-tls\") pod \"maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h\" (UID: \"66c2bf1c-95cc-43c0-9724-b89735374c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:11.079386 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.079362    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66c2bf1c-95cc-43c0-9724-b89735374c99-maas-api-tls\") pod \"maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h\" (UID: \"66c2bf1c-95cc-43c0-9724-b89735374c99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:11.115482 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.115449    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:11.167105 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.167072    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v"]
Sep 17 11:11:11.170934 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.170910    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v"
Sep 17 11:11:11.173973 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.173952    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-15da354a-dockercfg-zbv2j\""
Sep 17 11:11:11.174098 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.173977    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-15da354a\""
Sep 17 11:11:11.186739 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.186711    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v"]
Sep 17 11:11:11.267682 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.267602    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:11.278972 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.278942    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b9e9fa8-a361-4854-8274-4cd34b413ebd-plugins-config-volume\") pod \"payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v\" (UID: \"8b9e9fa8-a361-4854-8274-4cd34b413ebd\") " pod="openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v"
Sep 17 11:11:11.279052 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.279033    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nltmg\" (UniqueName: \"kubernetes.io/projected/8b9e9fa8-a361-4854-8274-4cd34b413ebd-kube-api-access-nltmg\") pod \"payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v\" (UID: \"8b9e9fa8-a361-4854-8274-4cd34b413ebd\") " pod="openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v"
Sep 17 11:11:11.380468 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.380433    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b9e9fa8-a361-4854-8274-4cd34b413ebd-plugins-config-volume\") pod \"payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v\" (UID: \"8b9e9fa8-a361-4854-8274-4cd34b413ebd\") " pod="openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v"
Sep 17 11:11:11.380638 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.380479    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nltmg\" (UniqueName: \"kubernetes.io/projected/8b9e9fa8-a361-4854-8274-4cd34b413ebd-kube-api-access-nltmg\") pod \"payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v\" (UID: \"8b9e9fa8-a361-4854-8274-4cd34b413ebd\") " pod="openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v"
Sep 17 11:11:11.381143 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.381123    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b9e9fa8-a361-4854-8274-4cd34b413ebd-plugins-config-volume\") pod \"payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v\" (UID: \"8b9e9fa8-a361-4854-8274-4cd34b413ebd\") " pod="openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v"
Sep 17 11:11:11.393153 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.393125    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nltmg\" (UniqueName: \"kubernetes.io/projected/8b9e9fa8-a361-4854-8274-4cd34b413ebd-kube-api-access-nltmg\") pod \"payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v\" (UID: \"8b9e9fa8-a361-4854-8274-4cd34b413ebd\") " pod="openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v"
Sep 17 11:11:11.481964 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.481931    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v"
Sep 17 11:11:11.582489 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.582423    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"]
Sep 17 11:11:11.585555 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:11:11.585529    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c2bf1c_95cc_43c0_9724_b89735374c99.slice/crio-23918c97aaf5ec7931fd928116503c82d41e6ec5d6e618f9b49990a045cff363 WatchSource:0}: Error finding container 23918c97aaf5ec7931fd928116503c82d41e6ec5d6e618f9b49990a045cff363: Status 404 returned error can't find the container with id 23918c97aaf5ec7931fd928116503c82d41e6ec5d6e618f9b49990a045cff363
Sep 17 11:11:11.587058 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.587036    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 11:11:11.754772 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.754749    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v"]
Sep 17 11:11:11.756561 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:11:11.756532    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9e9fa8_a361_4854_8274_4cd34b413ebd.slice/crio-b23fb3dccd2bf19433613e054318503e5a6f72d8a42e057c4d3d59c377cce516 WatchSource:0}: Error finding container b23fb3dccd2bf19433613e054318503e5a6f72d8a42e057c4d3d59c377cce516: Status 404 returned error can't find the container with id b23fb3dccd2bf19433613e054318503e5a6f72d8a42e057c4d3d59c377cce516
Sep 17 11:11:11.834798 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.834764    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:11.938424 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.938382    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h" event={"ID":"66c2bf1c-95cc-43c0-9724-b89735374c99","Type":"ContainerStarted","Data":"23918c97aaf5ec7931fd928116503c82d41e6ec5d6e618f9b49990a045cff363"}
Sep 17 11:11:11.940017 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.939993    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v" event={"ID":"8b9e9fa8-a361-4854-8274-4cd34b413ebd","Type":"ContainerStarted","Data":"f37ea5fe75fed405eeb8faf69f7a9ccbe262b484f574688555023e868798b359"}
Sep 17 11:11:11.940134 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.940035    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v" event={"ID":"8b9e9fa8-a361-4854-8274-4cd34b413ebd","Type":"ContainerStarted","Data":"b23fb3dccd2bf19433613e054318503e5a6f72d8a42e057c4d3d59c377cce516"}
Sep 17 11:11:11.940134 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.940104    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v"
Sep 17 11:11:11.980964 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:11.980904    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v" podStartSLOduration=0.980889208 podStartE2EDuration="980.889208ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:11:11.977997348 +0000 UTC m=+1066.076042914" watchObservedRunningTime="2026-09-17 11:11:11.980889208 +0000 UTC m=+1066.078934774"
Sep 17 11:11:12.678206 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:12.678173    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:12.946688 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:12.946590    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h" event={"ID":"66c2bf1c-95cc-43c0-9724-b89735374c99","Type":"ContainerStarted","Data":"f198c2288cf37a99702f36dfa27b5bc1b510d12aca689fbef082ff78532aebdb"}
Sep 17 11:11:12.946877 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:12.946837    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:12.971093 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:12.971045    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h" podStartSLOduration=2.548909198 podStartE2EDuration="2.971030834s" totalImagesPullingTime="422.121636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:11:12.970069143 +0000 UTC m=+1067.068114709" watchObservedRunningTime="2026-09-17 11:11:12.971030834 +0000 UTC m=+1067.069076396"
Sep 17 11:11:17.334848 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:17.334791    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:18.325043 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.325012    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:18.380485 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.380446    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4"]
Sep 17 11:11:18.380941 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.380699    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4" podUID="13422c64-a518-44f5-9772-c39c50d5962e" containerName="payload-pre-processing" containerID="cri-o://2ead94f3a16524d900c99ec94f246f2c2c620ca98b2f21be2156c8a45bb95957" gracePeriod=30
Sep 17 11:11:18.428843 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.428782    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:18.530361 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.530328    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:18.624535 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.624507    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:18.639385 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.639359    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4"
Sep 17 11:11:18.748307 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.748275    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13422c64-a518-44f5-9772-c39c50d5962e-plugins-config-volume\") pod \"13422c64-a518-44f5-9772-c39c50d5962e\" (UID: \"13422c64-a518-44f5-9772-c39c50d5962e\") "
Sep 17 11:11:18.748307 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.748311    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6b74\" (UniqueName: \"kubernetes.io/projected/13422c64-a518-44f5-9772-c39c50d5962e-kube-api-access-s6b74\") pod \"13422c64-a518-44f5-9772-c39c50d5962e\" (UID: \"13422c64-a518-44f5-9772-c39c50d5962e\") "
Sep 17 11:11:18.748642 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.748621    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13422c64-a518-44f5-9772-c39c50d5962e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13422c64-a518-44f5-9772-c39c50d5962e" (UID: "13422c64-a518-44f5-9772-c39c50d5962e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:11:18.750644 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.750621    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13422c64-a518-44f5-9772-c39c50d5962e-kube-api-access-s6b74" (OuterVolumeSpecName: "kube-api-access-s6b74") pod "13422c64-a518-44f5-9772-c39c50d5962e" (UID: "13422c64-a518-44f5-9772-c39c50d5962e"). InnerVolumeSpecName "kube-api-access-s6b74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:11:18.849942 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.849907    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13422c64-a518-44f5-9772-c39c50d5962e-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:18.849942 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.849936    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6b74\" (UniqueName: \"kubernetes.io/projected/13422c64-a518-44f5-9772-c39c50d5962e-kube-api-access-s6b74\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:18.957537 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.957442    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:18.988489 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.988455    2562 generic.go:356] "Generic (PLEG): container finished" podID="13422c64-a518-44f5-9772-c39c50d5962e" containerID="2ead94f3a16524d900c99ec94f246f2c2c620ca98b2f21be2156c8a45bb95957" exitCode=0
Sep 17 11:11:18.988691 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.988506    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4"
Sep 17 11:11:18.988691 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.988538    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4" event={"ID":"13422c64-a518-44f5-9772-c39c50d5962e","Type":"ContainerDied","Data":"2ead94f3a16524d900c99ec94f246f2c2c620ca98b2f21be2156c8a45bb95957"}
Sep 17 11:11:18.988691 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.988630    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4" event={"ID":"13422c64-a518-44f5-9772-c39c50d5962e","Type":"ContainerDied","Data":"406816b75c17facaf7e6d6ce93d39e078558bac3808dbfc46e8be7e6765acf94"}
Sep 17 11:11:18.988691 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.988660    2562 scope.go:117] "RemoveContainer" containerID="2ead94f3a16524d900c99ec94f246f2c2c620ca98b2f21be2156c8a45bb95957"
Sep 17 11:11:18.997601 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.997584    2562 scope.go:117] "RemoveContainer" containerID="2ead94f3a16524d900c99ec94f246f2c2c620ca98b2f21be2156c8a45bb95957"
Sep 17 11:11:18.997918 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:11:18.997898    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ead94f3a16524d900c99ec94f246f2c2c620ca98b2f21be2156c8a45bb95957\": container with ID starting with 2ead94f3a16524d900c99ec94f246f2c2c620ca98b2f21be2156c8a45bb95957 not found: ID does not exist" containerID="2ead94f3a16524d900c99ec94f246f2c2c620ca98b2f21be2156c8a45bb95957"
Sep 17 11:11:18.997988 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:18.997926    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ead94f3a16524d900c99ec94f246f2c2c620ca98b2f21be2156c8a45bb95957"} err="failed to get container status \"2ead94f3a16524d900c99ec94f246f2c2c620ca98b2f21be2156c8a45bb95957\": rpc error: code = NotFound desc = could not find container \"2ead94f3a16524d900c99ec94f246f2c2c620ca98b2f21be2156c8a45bb95957\": container with ID starting with 2ead94f3a16524d900c99ec94f246f2c2c620ca98b2f21be2156c8a45bb95957 not found: ID does not exist"
Sep 17 11:11:19.108674 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:19.108643    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4"]
Sep 17 11:11:19.125004 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:19.124976    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-79a70a-5fb9b6b9fd-vppr4"]
Sep 17 11:11:19.142546 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:19.142506    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:19.472134 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:19.472099    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:20.355188 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:20.355159    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:20.456334 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:20.456305    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13422c64-a518-44f5-9772-c39c50d5962e" path="/var/lib/kubelet/pods/13422c64-a518-44f5-9772-c39c50d5962e/volumes"
Sep 17 11:11:21.153233 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:21.153190    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:22.075041 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:22.075008    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:22.948873 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:22.948846    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v"
Sep 17 11:11:23.951473 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:23.951441    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:24.958000 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:24.957959    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:25.717521 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:25.717489    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:25.821975 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:25.821947    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:25.993028 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:25.992952    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:26.966889 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:26.966857    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:28.041267 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:28.041237    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:28.883969 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:28.883936    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:29.764291 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:29.764257    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:30.528245 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:30.528213    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:36.683668 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:36.683617    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:37.448215 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.448181    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"]
Sep 17 11:11:37.448469 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.448398    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h" podUID="66c2bf1c-95cc-43c0-9724-b89735374c99" containerName="maas-api" containerID="cri-o://f198c2288cf37a99702f36dfa27b5bc1b510d12aca689fbef082ff78532aebdb" gracePeriod=30
Sep 17 11:11:37.498982 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.498950    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:37.537023 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.536986    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v"]
Sep 17 11:11:37.537247 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.537224    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v" podUID="8b9e9fa8-a361-4854-8274-4cd34b413ebd" containerName="payload-processing" containerID="cri-o://f37ea5fe75fed405eeb8faf69f7a9ccbe262b484f574688555023e868798b359" gracePeriod=30
Sep 17 11:11:37.623092 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.623057    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:37.686001 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.685940    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"]
Sep 17 11:11:37.686515 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.686224    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq" podUID="52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0" containerName="maas-api" containerID="cri-o://8f17d107aaf18f316183f09a20f5bd25be847bb4243760829a710b23a17579c7" gracePeriod=30
Sep 17 11:11:37.717439 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.717414    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:37.758046 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.758010    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq"]
Sep 17 11:11:37.758272 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.758247    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq" podUID="3212535a-e929-48ff-9cef-5d9e7eaf889b" containerName="payload-processing" containerID="cri-o://e95c66f3e1ab6e145399cd0e0562d3071aa6e927d5be8ca043958c5e5c3ed10f" gracePeriod=30
Sep 17 11:11:37.775795 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.775554    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc"]
Sep 17 11:11:37.776280 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.776165    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc" podUID="0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad" containerName="payload-pre-processing" containerID="cri-o://1d77635fa7adf8c3c66852e7cf6bc0fd792949a0e8382cb23316d666f70d8350" gracePeriod=30
Sep 17 11:11:37.824349 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.824315    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:37.905038 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.905012    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v"
Sep 17 11:11:37.909759 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.909735    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:37.916873 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.916836    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:37.937085 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.937049    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b9e9fa8-a361-4854-8274-4cd34b413ebd-plugins-config-volume\") pod \"8b9e9fa8-a361-4854-8274-4cd34b413ebd\" (UID: \"8b9e9fa8-a361-4854-8274-4cd34b413ebd\") "
Sep 17 11:11:37.937265 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.937120    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c2bf1c-95cc-43c0-9724-b89735374c99-openshift-service-ca\") pod \"66c2bf1c-95cc-43c0-9724-b89735374c99\" (UID: \"66c2bf1c-95cc-43c0-9724-b89735374c99\") "
Sep 17 11:11:37.937265 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.937155    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nltmg\" (UniqueName: \"kubernetes.io/projected/8b9e9fa8-a361-4854-8274-4cd34b413ebd-kube-api-access-nltmg\") pod \"8b9e9fa8-a361-4854-8274-4cd34b413ebd\" (UID: \"8b9e9fa8-a361-4854-8274-4cd34b413ebd\") "
Sep 17 11:11:37.937265 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.937191    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljzg5\" (UniqueName: \"kubernetes.io/projected/66c2bf1c-95cc-43c0-9724-b89735374c99-kube-api-access-ljzg5\") pod \"66c2bf1c-95cc-43c0-9724-b89735374c99\" (UID: \"66c2bf1c-95cc-43c0-9724-b89735374c99\") "
Sep 17 11:11:37.937437 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.937276    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66c2bf1c-95cc-43c0-9724-b89735374c99-maas-api-tls\") pod \"66c2bf1c-95cc-43c0-9724-b89735374c99\" (UID: \"66c2bf1c-95cc-43c0-9724-b89735374c99\") "
Sep 17 11:11:37.937437 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.937350    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66c2bf1c-95cc-43c0-9724-b89735374c99-metrics-certs\") pod \"66c2bf1c-95cc-43c0-9724-b89735374c99\" (UID: \"66c2bf1c-95cc-43c0-9724-b89735374c99\") "
Sep 17 11:11:37.944631 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.937666    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9e9fa8-a361-4854-8274-4cd34b413ebd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b9e9fa8-a361-4854-8274-4cd34b413ebd" (UID: "8b9e9fa8-a361-4854-8274-4cd34b413ebd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:11:37.944631 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.938261    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c2bf1c-95cc-43c0-9724-b89735374c99-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66c2bf1c-95cc-43c0-9724-b89735374c99" (UID: "66c2bf1c-95cc-43c0-9724-b89735374c99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:11:37.944927 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.944675    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c2bf1c-95cc-43c0-9724-b89735374c99-kube-api-access-ljzg5" (OuterVolumeSpecName: "kube-api-access-ljzg5") pod "66c2bf1c-95cc-43c0-9724-b89735374c99" (UID: "66c2bf1c-95cc-43c0-9724-b89735374c99"). InnerVolumeSpecName "kube-api-access-ljzg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:11:37.945865 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.945307    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c2bf1c-95cc-43c0-9724-b89735374c99-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66c2bf1c-95cc-43c0-9724-b89735374c99" (UID: "66c2bf1c-95cc-43c0-9724-b89735374c99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:11:37.946634 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.946594    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c2bf1c-95cc-43c0-9724-b89735374c99-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "66c2bf1c-95cc-43c0-9724-b89735374c99" (UID: "66c2bf1c-95cc-43c0-9724-b89735374c99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:11:37.947270 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:37.947233    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9e9fa8-a361-4854-8274-4cd34b413ebd-kube-api-access-nltmg" (OuterVolumeSpecName: "kube-api-access-nltmg") pod "8b9e9fa8-a361-4854-8274-4cd34b413ebd" (UID: "8b9e9fa8-a361-4854-8274-4cd34b413ebd"). InnerVolumeSpecName "kube-api-access-nltmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:11:38.013585 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.013560    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:11:38.018321 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.018183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:38.038466 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.038433    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-openshift-service-ca\") pod \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\" (UID: \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\") "
Sep 17 11:11:38.038644 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.038524    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2lp2\" (UniqueName: \"kubernetes.io/projected/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-kube-api-access-s2lp2\") pod \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\" (UID: \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\") "
Sep 17 11:11:38.038644 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.038588    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-metrics-certs\") pod \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\" (UID: \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\") "
Sep 17 11:11:38.038761 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.038658    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-maas-api-tls\") pod \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\" (UID: \"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0\") "
Sep 17 11:11:38.039082 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.039024    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c2bf1c-95cc-43c0-9724-b89735374c99-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:38.039082 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.039051    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nltmg\" (UniqueName: \"kubernetes.io/projected/8b9e9fa8-a361-4854-8274-4cd34b413ebd-kube-api-access-nltmg\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:38.039082 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.039064    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljzg5\" (UniqueName: \"kubernetes.io/projected/66c2bf1c-95cc-43c0-9724-b89735374c99-kube-api-access-ljzg5\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:38.039082 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.039078    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66c2bf1c-95cc-43c0-9724-b89735374c99-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:38.039351 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.039091    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66c2bf1c-95cc-43c0-9724-b89735374c99-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:38.039351 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.039104    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b9e9fa8-a361-4854-8274-4cd34b413ebd-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:38.039653 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.039611    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0" (UID: "52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:11:38.041688 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.041650    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0" (UID: "52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:11:38.041688 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.041674    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0" (UID: "52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:11:38.044319 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.044287    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-kube-api-access-s2lp2" (OuterVolumeSpecName: "kube-api-access-s2lp2") pod "52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0" (UID: "52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0"). InnerVolumeSpecName "kube-api-access-s2lp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:11:38.059455 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.059432    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq"
Sep 17 11:11:38.085323 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.085299    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc"
Sep 17 11:11:38.099074 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.099046    2562 generic.go:356] "Generic (PLEG): container finished" podID="66c2bf1c-95cc-43c0-9724-b89735374c99" containerID="f198c2288cf37a99702f36dfa27b5bc1b510d12aca689fbef082ff78532aebdb" exitCode=0
Sep 17 11:11:38.099205 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.099090    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"
Sep 17 11:11:38.099205 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.099135    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h" event={"ID":"66c2bf1c-95cc-43c0-9724-b89735374c99","Type":"ContainerDied","Data":"f198c2288cf37a99702f36dfa27b5bc1b510d12aca689fbef082ff78532aebdb"}
Sep 17 11:11:38.099205 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.099181    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h" event={"ID":"66c2bf1c-95cc-43c0-9724-b89735374c99","Type":"ContainerDied","Data":"23918c97aaf5ec7931fd928116503c82d41e6ec5d6e618f9b49990a045cff363"}
Sep 17 11:11:38.099348 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.099208    2562 scope.go:117] "RemoveContainer" containerID="f198c2288cf37a99702f36dfa27b5bc1b510d12aca689fbef082ff78532aebdb"
Sep 17 11:11:38.100630 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.100598    2562 generic.go:356] "Generic (PLEG): container finished" podID="8b9e9fa8-a361-4854-8274-4cd34b413ebd" containerID="f37ea5fe75fed405eeb8faf69f7a9ccbe262b484f574688555023e868798b359" exitCode=0
Sep 17 11:11:38.100732 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.100638    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v"
Sep 17 11:11:38.100732 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.100682    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v" event={"ID":"8b9e9fa8-a361-4854-8274-4cd34b413ebd","Type":"ContainerDied","Data":"f37ea5fe75fed405eeb8faf69f7a9ccbe262b484f574688555023e868798b359"}
Sep 17 11:11:38.100875 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.100762    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v" event={"ID":"8b9e9fa8-a361-4854-8274-4cd34b413ebd","Type":"ContainerDied","Data":"b23fb3dccd2bf19433613e054318503e5a6f72d8a42e057c4d3d59c377cce516"}
Sep 17 11:11:38.102607 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.102583    2562 generic.go:356] "Generic (PLEG): container finished" podID="0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad" containerID="1d77635fa7adf8c3c66852e7cf6bc0fd792949a0e8382cb23316d666f70d8350" exitCode=0
Sep 17 11:11:38.102717 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.102626    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc"
Sep 17 11:11:38.102717 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.102686    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc" event={"ID":"0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad","Type":"ContainerDied","Data":"1d77635fa7adf8c3c66852e7cf6bc0fd792949a0e8382cb23316d666f70d8350"}
Sep 17 11:11:38.102866 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.102777    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc" event={"ID":"0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad","Type":"ContainerDied","Data":"29076d80b8b70626b8a2406a4403ded0cedaefcbc325fb2b245826620f9a92ff"}
Sep 17 11:11:38.104922 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.104808    2562 generic.go:356] "Generic (PLEG): container finished" podID="3212535a-e929-48ff-9cef-5d9e7eaf889b" containerID="e95c66f3e1ab6e145399cd0e0562d3071aa6e927d5be8ca043958c5e5c3ed10f" exitCode=0
Sep 17 11:11:38.104922 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.104897    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq"
Sep 17 11:11:38.104922 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.104901    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq" event={"ID":"3212535a-e929-48ff-9cef-5d9e7eaf889b","Type":"ContainerDied","Data":"e95c66f3e1ab6e145399cd0e0562d3071aa6e927d5be8ca043958c5e5c3ed10f"}
Sep 17 11:11:38.105704 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.104967    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq" event={"ID":"3212535a-e929-48ff-9cef-5d9e7eaf889b","Type":"ContainerDied","Data":"5378c449565dafd897ac3528377375c6ada4d7591dd6772219cb9a75981bc8cf"}
Sep 17 11:11:38.107113 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.107082    2562 generic.go:356] "Generic (PLEG): container finished" podID="52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0" containerID="8f17d107aaf18f316183f09a20f5bd25be847bb4243760829a710b23a17579c7" exitCode=0
Sep 17 11:11:38.107213 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.107120    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq" event={"ID":"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0","Type":"ContainerDied","Data":"8f17d107aaf18f316183f09a20f5bd25be847bb4243760829a710b23a17579c7"}
Sep 17 11:11:38.107213 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.107156    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq" event={"ID":"52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0","Type":"ContainerDied","Data":"a3f588662c189d28a24f784287a8a37cbb10cc1ee7e2b11a327aaada4e0536b8"}
Sep 17 11:11:38.107213 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.107120    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"
Sep 17 11:11:38.111464 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.111442    2562 scope.go:117] "RemoveContainer" containerID="f198c2288cf37a99702f36dfa27b5bc1b510d12aca689fbef082ff78532aebdb"
Sep 17 11:11:38.111808 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:11:38.111784    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f198c2288cf37a99702f36dfa27b5bc1b510d12aca689fbef082ff78532aebdb\": container with ID starting with f198c2288cf37a99702f36dfa27b5bc1b510d12aca689fbef082ff78532aebdb not found: ID does not exist" containerID="f198c2288cf37a99702f36dfa27b5bc1b510d12aca689fbef082ff78532aebdb"
Sep 17 11:11:38.111808 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.111853    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f198c2288cf37a99702f36dfa27b5bc1b510d12aca689fbef082ff78532aebdb"} err="failed to get container status \"f198c2288cf37a99702f36dfa27b5bc1b510d12aca689fbef082ff78532aebdb\": rpc error: code = NotFound desc = could not find container \"f198c2288cf37a99702f36dfa27b5bc1b510d12aca689fbef082ff78532aebdb\": container with ID starting with f198c2288cf37a99702f36dfa27b5bc1b510d12aca689fbef082ff78532aebdb not found: ID does not exist"
Sep 17 11:11:38.111808 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.111880    2562 scope.go:117] "RemoveContainer" containerID="f37ea5fe75fed405eeb8faf69f7a9ccbe262b484f574688555023e868798b359"
Sep 17 11:11:38.123497 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.123475    2562 scope.go:117] "RemoveContainer" containerID="f37ea5fe75fed405eeb8faf69f7a9ccbe262b484f574688555023e868798b359"
Sep 17 11:11:38.123867 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:11:38.123844    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f37ea5fe75fed405eeb8faf69f7a9ccbe262b484f574688555023e868798b359\": container with ID starting with f37ea5fe75fed405eeb8faf69f7a9ccbe262b484f574688555023e868798b359 not found: ID does not exist" containerID="f37ea5fe75fed405eeb8faf69f7a9ccbe262b484f574688555023e868798b359"
Sep 17 11:11:38.123948 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.123878    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f37ea5fe75fed405eeb8faf69f7a9ccbe262b484f574688555023e868798b359"} err="failed to get container status \"f37ea5fe75fed405eeb8faf69f7a9ccbe262b484f574688555023e868798b359\": rpc error: code = NotFound desc = could not find container \"f37ea5fe75fed405eeb8faf69f7a9ccbe262b484f574688555023e868798b359\": container with ID starting with f37ea5fe75fed405eeb8faf69f7a9ccbe262b484f574688555023e868798b359 not found: ID does not exist"
Sep 17 11:11:38.123948 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.123901    2562 scope.go:117] "RemoveContainer" containerID="1d77635fa7adf8c3c66852e7cf6bc0fd792949a0e8382cb23316d666f70d8350"
Sep 17 11:11:38.133664 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.133645    2562 scope.go:117] "RemoveContainer" containerID="1d77635fa7adf8c3c66852e7cf6bc0fd792949a0e8382cb23316d666f70d8350"
Sep 17 11:11:38.134103 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:11:38.134057    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d77635fa7adf8c3c66852e7cf6bc0fd792949a0e8382cb23316d666f70d8350\": container with ID starting with 1d77635fa7adf8c3c66852e7cf6bc0fd792949a0e8382cb23316d666f70d8350 not found: ID does not exist" containerID="1d77635fa7adf8c3c66852e7cf6bc0fd792949a0e8382cb23316d666f70d8350"
Sep 17 11:11:38.134233 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.134097    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d77635fa7adf8c3c66852e7cf6bc0fd792949a0e8382cb23316d666f70d8350"} err="failed to get container status \"1d77635fa7adf8c3c66852e7cf6bc0fd792949a0e8382cb23316d666f70d8350\": rpc error: code = NotFound desc = could not find container \"1d77635fa7adf8c3c66852e7cf6bc0fd792949a0e8382cb23316d666f70d8350\": container with ID starting with 1d77635fa7adf8c3c66852e7cf6bc0fd792949a0e8382cb23316d666f70d8350 not found: ID does not exist"
Sep 17 11:11:38.134233 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.134122    2562 scope.go:117] "RemoveContainer" containerID="e95c66f3e1ab6e145399cd0e0562d3071aa6e927d5be8ca043958c5e5c3ed10f"
Sep 17 11:11:38.140384 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.140363    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad-plugins-config-volume\") pod \"0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad\" (UID: \"0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad\") "
Sep 17 11:11:38.140514 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.140458    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3212535a-e929-48ff-9cef-5d9e7eaf889b-plugins-config-volume\") pod \"3212535a-e929-48ff-9cef-5d9e7eaf889b\" (UID: \"3212535a-e929-48ff-9cef-5d9e7eaf889b\") "
Sep 17 11:11:38.140514 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.140486    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmdj8\" (UniqueName: \"kubernetes.io/projected/3212535a-e929-48ff-9cef-5d9e7eaf889b-kube-api-access-tmdj8\") pod \"3212535a-e929-48ff-9cef-5d9e7eaf889b\" (UID: \"3212535a-e929-48ff-9cef-5d9e7eaf889b\") "
Sep 17 11:11:38.140637 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.140573    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmrtv\" (UniqueName: \"kubernetes.io/projected/0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad-kube-api-access-mmrtv\") pod \"0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad\" (UID: \"0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad\") "
Sep 17 11:11:38.140886 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.140848    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad" (UID: "0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:11:38.140999 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.140966    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3212535a-e929-48ff-9cef-5d9e7eaf889b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3212535a-e929-48ff-9cef-5d9e7eaf889b" (UID: "3212535a-e929-48ff-9cef-5d9e7eaf889b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:11:38.141079 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.140973    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:38.141079 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.141018    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2lp2\" (UniqueName: \"kubernetes.io/projected/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-kube-api-access-s2lp2\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:38.141079 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.141029    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:38.141079 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.141038    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:38.141079 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.141047    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:38.142957 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.142937    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad-kube-api-access-mmrtv" (OuterVolumeSpecName: "kube-api-access-mmrtv") pod "0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad" (UID: "0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad"). InnerVolumeSpecName "kube-api-access-mmrtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:11:38.143200 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.143181    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3212535a-e929-48ff-9cef-5d9e7eaf889b-kube-api-access-tmdj8" (OuterVolumeSpecName: "kube-api-access-tmdj8") pod "3212535a-e929-48ff-9cef-5d9e7eaf889b" (UID: "3212535a-e929-48ff-9cef-5d9e7eaf889b"). InnerVolumeSpecName "kube-api-access-tmdj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:11:38.144280 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.144259    2562 scope.go:117] "RemoveContainer" containerID="e95c66f3e1ab6e145399cd0e0562d3071aa6e927d5be8ca043958c5e5c3ed10f"
Sep 17 11:11:38.144716 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:11:38.144687    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95c66f3e1ab6e145399cd0e0562d3071aa6e927d5be8ca043958c5e5c3ed10f\": container with ID starting with e95c66f3e1ab6e145399cd0e0562d3071aa6e927d5be8ca043958c5e5c3ed10f not found: ID does not exist" containerID="e95c66f3e1ab6e145399cd0e0562d3071aa6e927d5be8ca043958c5e5c3ed10f"
Sep 17 11:11:38.144809 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.144725    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95c66f3e1ab6e145399cd0e0562d3071aa6e927d5be8ca043958c5e5c3ed10f"} err="failed to get container status \"e95c66f3e1ab6e145399cd0e0562d3071aa6e927d5be8ca043958c5e5c3ed10f\": rpc error: code = NotFound desc = could not find container \"e95c66f3e1ab6e145399cd0e0562d3071aa6e927d5be8ca043958c5e5c3ed10f\": container with ID starting with e95c66f3e1ab6e145399cd0e0562d3071aa6e927d5be8ca043958c5e5c3ed10f not found: ID does not exist"
Sep 17 11:11:38.144809 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.144749    2562 scope.go:117] "RemoveContainer" containerID="8f17d107aaf18f316183f09a20f5bd25be847bb4243760829a710b23a17579c7"
Sep 17 11:11:38.152857 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.152838    2562 scope.go:117] "RemoveContainer" containerID="8f17d107aaf18f316183f09a20f5bd25be847bb4243760829a710b23a17579c7"
Sep 17 11:11:38.153146 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:11:38.153126    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f17d107aaf18f316183f09a20f5bd25be847bb4243760829a710b23a17579c7\": container with ID starting with 8f17d107aaf18f316183f09a20f5bd25be847bb4243760829a710b23a17579c7 not found: ID does not exist" containerID="8f17d107aaf18f316183f09a20f5bd25be847bb4243760829a710b23a17579c7"
Sep 17 11:11:38.153224 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.153149    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f17d107aaf18f316183f09a20f5bd25be847bb4243760829a710b23a17579c7"} err="failed to get container status \"8f17d107aaf18f316183f09a20f5bd25be847bb4243760829a710b23a17579c7\": rpc error: code = NotFound desc = could not find container \"8f17d107aaf18f316183f09a20f5bd25be847bb4243760829a710b23a17579c7\": container with ID starting with 8f17d107aaf18f316183f09a20f5bd25be847bb4243760829a710b23a17579c7 not found: ID does not exist"
Sep 17 11:11:38.187722 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.187693    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"]
Sep 17 11:11:38.194903 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.194870    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c10e0b-55dd6f65f5-hxszq"]
Sep 17 11:11:38.228965 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.228885    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"]
Sep 17 11:11:38.232999 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.232976    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15da354a-84b5644f8d-dcc4h"]
Sep 17 11:11:38.241759 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.241737    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmrtv\" (UniqueName: \"kubernetes.io/projected/0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad-kube-api-access-mmrtv\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:38.241759 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.241762    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3212535a-e929-48ff-9cef-5d9e7eaf889b-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:38.241928 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.241772    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmdj8\" (UniqueName: \"kubernetes.io/projected/3212535a-e929-48ff-9cef-5d9e7eaf889b-kube-api-access-tmdj8\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:11:38.254430 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.254400    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v"]
Sep 17 11:11:38.262000 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.261977    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15da354a-8469b6cfdc-n5b9v"]
Sep 17 11:11:38.358873 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.358795    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:38.437960 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.437920    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc"]
Sep 17 11:11:38.448518 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.448489    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c10e0b-7c5677b8d6-fsmcc"]
Sep 17 11:11:38.455580 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.455555    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad" path="/var/lib/kubelet/pods/0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad/volumes"
Sep 17 11:11:38.455971 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.455957    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0" path="/var/lib/kubelet/pods/52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0/volumes"
Sep 17 11:11:38.456312 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.456301    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66c2bf1c-95cc-43c0-9724-b89735374c99" path="/var/lib/kubelet/pods/66c2bf1c-95cc-43c0-9724-b89735374c99/volumes"
Sep 17 11:11:38.456920 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.456907    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b9e9fa8-a361-4854-8274-4cd34b413ebd" path="/var/lib/kubelet/pods/8b9e9fa8-a361-4854-8274-4cd34b413ebd/volumes"
Sep 17 11:11:38.465790 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.465771    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq"]
Sep 17 11:11:38.484129 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.484065    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c10e0b-85b55dc9b5-cx2qq"]
Sep 17 11:11:38.670098 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.670060    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:38.772014 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:38.771932    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:39.027838 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:39.027800    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:39.462030 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:39.461947    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:39.583227 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:39.583199    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:39.828396 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:39.828362    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:40.238318 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:40.238235    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:40.456374 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:40.456344    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3212535a-e929-48ff-9cef-5d9e7eaf889b" path="/var/lib/kubelet/pods/3212535a-e929-48ff-9cef-5d9e7eaf889b/volumes"
Sep 17 11:11:40.868547 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:40.868516    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:41.330358 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:41.330321    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:11:42.005238 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:42.005208    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:42.445700 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:42.445668    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:42.582309 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:42.582273    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:43.191671 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:43.191640    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:44.235301 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235265    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"]
Sep 17 11:11:44.235754 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235665    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13422c64-a518-44f5-9772-c39c50d5962e" containerName="payload-pre-processing"
Sep 17 11:11:44.235754 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235677    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="13422c64-a518-44f5-9772-c39c50d5962e" containerName="payload-pre-processing"
Sep 17 11:11:44.235754 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235687    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0" containerName="maas-api"
Sep 17 11:11:44.235754 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235692    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0" containerName="maas-api"
Sep 17 11:11:44.235754 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235706    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9e9fa8-a361-4854-8274-4cd34b413ebd" containerName="payload-processing"
Sep 17 11:11:44.235754 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235715    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9e9fa8-a361-4854-8274-4cd34b413ebd" containerName="payload-processing"
Sep 17 11:11:44.235754 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235728    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c2bf1c-95cc-43c0-9724-b89735374c99" containerName="maas-api"
Sep 17 11:11:44.235754 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235734    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c2bf1c-95cc-43c0-9724-b89735374c99" containerName="maas-api"
Sep 17 11:11:44.235754 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235749    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3212535a-e929-48ff-9cef-5d9e7eaf889b" containerName="payload-processing"
Sep 17 11:11:44.235754 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235754    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3212535a-e929-48ff-9cef-5d9e7eaf889b" containerName="payload-processing"
Sep 17 11:11:44.235754 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235762    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad" containerName="payload-pre-processing"
Sep 17 11:11:44.236397 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235767    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad" containerName="payload-pre-processing"
Sep 17 11:11:44.236397 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235864    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b9e9fa8-a361-4854-8274-4cd34b413ebd" containerName="payload-processing"
Sep 17 11:11:44.236397 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235875    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="13422c64-a518-44f5-9772-c39c50d5962e" containerName="payload-pre-processing"
Sep 17 11:11:44.236397 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235884    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="52fa6f4b-6663-4ea7-a21c-efb2a32ba3c0" containerName="maas-api"
Sep 17 11:11:44.236397 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235894    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c2bf1c-95cc-43c0-9724-b89735374c99" containerName="maas-api"
Sep 17 11:11:44.236397 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235903    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3212535a-e929-48ff-9cef-5d9e7eaf889b" containerName="payload-processing"
Sep 17 11:11:44.236397 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.235910    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c86dcfb-ff1e-4063-bd5e-e12d0e8a75ad" containerName="payload-pre-processing"
Sep 17 11:11:44.240806 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.240784    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:11:44.243689 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.243666    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-15f9c946-serving-cert\""
Sep 17 11:11:44.243932 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.243918    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gvplj\""
Sep 17 11:11:44.245060 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.245040    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 11:11:44.245177 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.245062    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 17 11:11:44.245177 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.245078    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 17 11:11:44.252256 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.252236    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"]
Sep 17 11:11:44.278855 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.278792    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:44.293658 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.293630    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-metrics-certs\") pod \"maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp\" (UID: \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:11:44.293792 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.293705    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-maas-api-tls\") pod \"maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp\" (UID: \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:11:44.293792 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.293725    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-openshift-service-ca\") pod \"maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp\" (UID: \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:11:44.293792 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.293760    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvz5q\" (UniqueName: \"kubernetes.io/projected/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-kube-api-access-vvz5q\") pod \"maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp\" (UID: \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:11:44.381328 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.381296    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:44.394508 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.394481    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-maas-api-tls\") pod \"maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp\" (UID: \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:11:44.394508 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.394511    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-openshift-service-ca\") pod \"maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp\" (UID: \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:11:44.394677 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.394550    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvz5q\" (UniqueName: \"kubernetes.io/projected/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-kube-api-access-vvz5q\") pod \"maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp\" (UID: \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:11:44.394677 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.394577    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-metrics-certs\") pod \"maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp\" (UID: \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:11:44.394677 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:11:44.394620    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-serving-cert: secret "maas-api-e2e-derive-15f9c946-serving-cert" not found
Sep 17 11:11:44.394806 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:11:44.394680    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-maas-api-tls podName:e58d8aae-4bd4-4d67-bd6b-0b27070383d6 nodeName:}" failed. No retries permitted until 2026-09-17 11:11:44.894665131 +0000 UTC m=+1098.992710678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-maas-api-tls") pod "maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp" (UID: "e58d8aae-4bd4-4d67-bd6b-0b27070383d6") : secret "maas-api-e2e-derive-15f9c946-serving-cert" not found
Sep 17 11:11:44.395268 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.395251    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-openshift-service-ca\") pod \"maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp\" (UID: \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:11:44.397175 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.397159    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-metrics-certs\") pod \"maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp\" (UID: \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:11:44.405862 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.405834    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvz5q\" (UniqueName: \"kubernetes.io/projected/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-kube-api-access-vvz5q\") pod \"maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp\" (UID: \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:11:44.594472 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.594440    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:44.808620 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.808590    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:44.900554 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.900471    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-maas-api-tls\") pod \"maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp\" (UID: \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:11:44.903054 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.903028    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-maas-api-tls\") pod \"maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp\" (UID: \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:11:44.903182 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.903107    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:44.942475 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.942440    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v"]
Sep 17 11:11:44.946897 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.946880    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v"
Sep 17 11:11:44.949515 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.949493    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-15f9c946\""
Sep 17 11:11:44.949610 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.949590    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-15f9c946-dockercfg-qhz92\""
Sep 17 11:11:44.962924 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:44.962899    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v"]
Sep 17 11:11:45.000988 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:45.000953    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v55q\" (UniqueName: \"kubernetes.io/projected/0e084ea3-7600-4ac5-9fbe-7cdae76c6c34-kube-api-access-2v55q\") pod \"payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v\" (UID: \"0e084ea3-7600-4ac5-9fbe-7cdae76c6c34\") " pod="openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v"
Sep 17 11:11:45.001157 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:45.001096    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e084ea3-7600-4ac5-9fbe-7cdae76c6c34-plugins-config-volume\") pod \"payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v\" (UID: \"0e084ea3-7600-4ac5-9fbe-7cdae76c6c34\") " pod="openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v"
Sep 17 11:11:45.101623 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:45.101587    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e084ea3-7600-4ac5-9fbe-7cdae76c6c34-plugins-config-volume\") pod \"payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v\" (UID: \"0e084ea3-7600-4ac5-9fbe-7cdae76c6c34\") " pod="openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v"
Sep 17 11:11:45.101857 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:45.101650    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v55q\" (UniqueName: \"kubernetes.io/projected/0e084ea3-7600-4ac5-9fbe-7cdae76c6c34-kube-api-access-2v55q\") pod \"payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v\" (UID: \"0e084ea3-7600-4ac5-9fbe-7cdae76c6c34\") " pod="openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v"
Sep 17 11:11:45.102374 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:45.102350    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e084ea3-7600-4ac5-9fbe-7cdae76c6c34-plugins-config-volume\") pod \"payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v\" (UID: \"0e084ea3-7600-4ac5-9fbe-7cdae76c6c34\") " pod="openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v"
Sep 17 11:11:45.110869 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:45.110844    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v55q\" (UniqueName: \"kubernetes.io/projected/0e084ea3-7600-4ac5-9fbe-7cdae76c6c34-kube-api-access-2v55q\") pod \"payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v\" (UID: \"0e084ea3-7600-4ac5-9fbe-7cdae76c6c34\") " pod="openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v"
Sep 17 11:11:45.151958 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:45.151880    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:11:45.257338 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:45.257301    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v"
Sep 17 11:11:45.427465 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:45.427438    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"]
Sep 17 11:11:45.428613 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:11:45.428592    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58d8aae_4bd4_4d67_bd6b_0b27070383d6.slice/crio-080ffc2f03f964e71213932e03ab36264180c0f419e8cacc50373dac3575d62b WatchSource:0}: Error finding container 080ffc2f03f964e71213932e03ab36264180c0f419e8cacc50373dac3575d62b: Status 404 returned error can't find the container with id 080ffc2f03f964e71213932e03ab36264180c0f419e8cacc50373dac3575d62b
Sep 17 11:11:45.539429 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:45.539401    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v"]
Sep 17 11:11:45.541094 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:11:45.541065    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e084ea3_7600_4ac5_9fbe_7cdae76c6c34.slice/crio-985e84ccfd1c8ac04bb1f9a1242ca12be76de0b3718454e113ed8f18b11d4ec7 WatchSource:0}: Error finding container 985e84ccfd1c8ac04bb1f9a1242ca12be76de0b3718454e113ed8f18b11d4ec7: Status 404 returned error can't find the container with id 985e84ccfd1c8ac04bb1f9a1242ca12be76de0b3718454e113ed8f18b11d4ec7
Sep 17 11:11:45.966778 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:45.966709    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:46.068124 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:46.068090    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:46.159331 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:46.159291    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v" event={"ID":"0e084ea3-7600-4ac5-9fbe-7cdae76c6c34","Type":"ContainerStarted","Data":"84372e3ad95fc177d7b22565cf7bcccf36c83185553a71acfb2afa83a612f34d"}
Sep 17 11:11:46.159520 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:46.159348    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v" event={"ID":"0e084ea3-7600-4ac5-9fbe-7cdae76c6c34","Type":"ContainerStarted","Data":"985e84ccfd1c8ac04bb1f9a1242ca12be76de0b3718454e113ed8f18b11d4ec7"}
Sep 17 11:11:46.159520 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:46.159377    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v"
Sep 17 11:11:46.160829 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:46.160783    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp" event={"ID":"e58d8aae-4bd4-4d67-bd6b-0b27070383d6","Type":"ContainerStarted","Data":"d386835140de7eb7b34d1283460f47f72b047d8dd66d95298850e571503b7f72"}
Sep 17 11:11:46.160946 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:46.160856    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp" event={"ID":"e58d8aae-4bd4-4d67-bd6b-0b27070383d6","Type":"ContainerStarted","Data":"080ffc2f03f964e71213932e03ab36264180c0f419e8cacc50373dac3575d62b"}
Sep 17 11:11:46.160946 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:46.160879    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:11:46.184618 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:46.184570    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v" podStartSLOduration=2.184555398 podStartE2EDuration="2.184555398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:11:46.183650939 +0000 UTC m=+1100.281696504" watchObservedRunningTime="2026-09-17 11:11:46.184555398 +0000 UTC m=+1100.282600964"
Sep 17 11:11:46.212544 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:46.212472    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp" podStartSLOduration=1.935910577 podStartE2EDuration="2.212450827s" totalImagesPullingTime="276.54025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:11:46.208937303 +0000 UTC m=+1100.306982882" watchObservedRunningTime="2026-09-17 11:11:46.212450827 +0000 UTC m=+1100.310496395"
Sep 17 11:11:46.259955 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:46.259866    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:46.830693 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:46.830661    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:47.364364 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:47.364332    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:47.825547 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:47.825517    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:49.495181 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:49.495149    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:50.035257 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:50.035224    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:50.253285 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:50.253252    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:50.670144 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:50.670110    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:51.551603 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:51.551560    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:52.171782 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.171754    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:11:52.306236 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.306203    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"]
Sep 17 11:11:52.310033 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.310013    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:11:52.312980 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.312957    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-026f86-serving-cert\""
Sep 17 11:11:52.325212 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.325180    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"]
Sep 17 11:11:52.364509 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.364474    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:52.374556 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.374530    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e33cfb7-8379-4eac-bf61-e2294e2563ce-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r\" (UID: \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:11:52.374666 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.374588    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpprh\" (UniqueName: \"kubernetes.io/projected/9e33cfb7-8379-4eac-bf61-e2294e2563ce-kube-api-access-tpprh\") pod \"maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r\" (UID: \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:11:52.374720 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.374704    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e33cfb7-8379-4eac-bf61-e2294e2563ce-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r\" (UID: \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:11:52.374759 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.374745    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e33cfb7-8379-4eac-bf61-e2294e2563ce-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r\" (UID: \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:11:52.470179 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.470094    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:52.475362 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.475332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e33cfb7-8379-4eac-bf61-e2294e2563ce-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r\" (UID: \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:11:52.475485 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.475373    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e33cfb7-8379-4eac-bf61-e2294e2563ce-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r\" (UID: \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:11:52.475485 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.475397    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e33cfb7-8379-4eac-bf61-e2294e2563ce-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r\" (UID: \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:11:52.475485 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.475438    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpprh\" (UniqueName: \"kubernetes.io/projected/9e33cfb7-8379-4eac-bf61-e2294e2563ce-kube-api-access-tpprh\") pod \"maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r\" (UID: \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:11:52.475599 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:11:52.475486    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-serving-cert: secret "maas-api-e2e-shared-a-w2-026f86-serving-cert" not found
Sep 17 11:11:52.475599 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:11:52.475567    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e33cfb7-8379-4eac-bf61-e2294e2563ce-maas-api-tls podName:9e33cfb7-8379-4eac-bf61-e2294e2563ce nodeName:}" failed. No retries permitted until 2026-09-17 11:11:52.975549287 +0000 UTC m=+1107.073594845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e33cfb7-8379-4eac-bf61-e2294e2563ce-maas-api-tls") pod "maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r" (UID: "9e33cfb7-8379-4eac-bf61-e2294e2563ce") : secret "maas-api-e2e-shared-a-w2-026f86-serving-cert" not found
Sep 17 11:11:52.476070 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.476051    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e33cfb7-8379-4eac-bf61-e2294e2563ce-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r\" (UID: \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:11:52.477882 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.477863    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e33cfb7-8379-4eac-bf61-e2294e2563ce-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r\" (UID: \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:11:52.487987 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.487966    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpprh\" (UniqueName: \"kubernetes.io/projected/9e33cfb7-8379-4eac-bf61-e2294e2563ce-kube-api-access-tpprh\") pod \"maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r\" (UID: \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:11:52.563976 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.563943    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:52.855958 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.855913    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:52.952005 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.951956    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:52.980142 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.980108    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e33cfb7-8379-4eac-bf61-e2294e2563ce-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r\" (UID: \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:11:52.982626 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:52.982596    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e33cfb7-8379-4eac-bf61-e2294e2563ce-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r\" (UID: \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:11:53.221043 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:53.220949    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:11:53.494217 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:53.494190    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"]
Sep 17 11:11:53.496343 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:11:53.496320    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e33cfb7_8379_4eac_bf61_e2294e2563ce.slice/crio-ee270bc58616d66107725bb8fe1e7a2fbf5ec8dbc5fffb3daf994481a3b7819f WatchSource:0}: Error finding container ee270bc58616d66107725bb8fe1e7a2fbf5ec8dbc5fffb3daf994481a3b7819f: Status 404 returned error can't find the container with id ee270bc58616d66107725bb8fe1e7a2fbf5ec8dbc5fffb3daf994481a3b7819f
Sep 17 11:11:53.655088 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:53.655051    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:54.206555 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:54.206463    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r" event={"ID":"9e33cfb7-8379-4eac-bf61-e2294e2563ce","Type":"ContainerStarted","Data":"c1dd44b22a7711cec599831ea08482143de1d7e41a7de1bf186df3dac99e37f4"}
Sep 17 11:11:54.206555 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:54.206521    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r" event={"ID":"9e33cfb7-8379-4eac-bf61-e2294e2563ce","Type":"ContainerStarted","Data":"ee270bc58616d66107725bb8fe1e7a2fbf5ec8dbc5fffb3daf994481a3b7819f"}
Sep 17 11:11:54.206555 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:54.206549    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:11:54.224570 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:54.224522    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r" podStartSLOduration=1.891278937 podStartE2EDuration="2.224505201s" totalImagesPullingTime="333.226264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:11:54.224182694 +0000 UTC m=+1108.322228271" watchObservedRunningTime="2026-09-17 11:11:54.224505201 +0000 UTC m=+1108.322550750"
Sep 17 11:11:54.353132 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:54.353101    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:11:57.168695 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:11:57.168661    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v"
Sep 17 11:12:00.219172 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:00.219140    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:12:05.271555 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:05.271506    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:07.052913 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:07.052877    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:08.382587 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:08.382559    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:09.405023 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:09.404990    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:09.825526 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:09.825482    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:10.223048 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:10.222965    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:10.385829 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:10.385795    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:10.650013 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:10.649971    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:10.880962 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:10.880926    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:11.667291 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:11.667258    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:12.685929 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:12.685898    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:13.843941 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:13.843907    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"]
Sep 17 11:12:13.844369 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:13.844137    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp" podUID="e58d8aae-4bd4-4d67-bd6b-0b27070383d6" containerName="maas-api" containerID="cri-o://d386835140de7eb7b34d1283460f47f72b047d8dd66d95298850e571503b7f72" gracePeriod=30
Sep 17 11:12:13.888114 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:13.888066    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:13.944094 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:13.944060    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v"]
Sep 17 11:12:13.944292 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:13.944269    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v" podUID="0e084ea3-7600-4ac5-9fbe-7cdae76c6c34" containerName="payload-processing" containerID="cri-o://84372e3ad95fc177d7b22565cf7bcccf36c83185553a71acfb2afa83a612f34d" gracePeriod=30
Sep 17 11:12:14.017997 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.017964    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:14.108130 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.108062    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:12:14.115002 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.114976    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:14.189628 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.189604    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-metrics-certs\") pod \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\" (UID: \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\") "
Sep 17 11:12:14.189749 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.189637    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvz5q\" (UniqueName: \"kubernetes.io/projected/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-kube-api-access-vvz5q\") pod \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\" (UID: \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\") "
Sep 17 11:12:14.189749 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.189678    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-maas-api-tls\") pod \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\" (UID: \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\") "
Sep 17 11:12:14.189867 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.189762    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-openshift-service-ca\") pod \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\" (UID: \"e58d8aae-4bd4-4d67-bd6b-0b27070383d6\") "
Sep 17 11:12:14.190266 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.190240    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e58d8aae-4bd4-4d67-bd6b-0b27070383d6" (UID: "e58d8aae-4bd4-4d67-bd6b-0b27070383d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:12:14.191988 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.191967    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e58d8aae-4bd4-4d67-bd6b-0b27070383d6" (UID: "e58d8aae-4bd4-4d67-bd6b-0b27070383d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:12:14.192075 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.192025    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e58d8aae-4bd4-4d67-bd6b-0b27070383d6" (UID: "e58d8aae-4bd4-4d67-bd6b-0b27070383d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:12:14.192075 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.192055    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-kube-api-access-vvz5q" (OuterVolumeSpecName: "kube-api-access-vvz5q") pod "e58d8aae-4bd4-4d67-bd6b-0b27070383d6" (UID: "e58d8aae-4bd4-4d67-bd6b-0b27070383d6"). InnerVolumeSpecName "kube-api-access-vvz5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:12:14.194330 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.194313    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v"
Sep 17 11:12:14.255395 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.255362    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:14.290687 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.290657    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e084ea3-7600-4ac5-9fbe-7cdae76c6c34-plugins-config-volume\") pod \"0e084ea3-7600-4ac5-9fbe-7cdae76c6c34\" (UID: \"0e084ea3-7600-4ac5-9fbe-7cdae76c6c34\") "
Sep 17 11:12:14.290867 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.290693    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v55q\" (UniqueName: \"kubernetes.io/projected/0e084ea3-7600-4ac5-9fbe-7cdae76c6c34-kube-api-access-2v55q\") pod \"0e084ea3-7600-4ac5-9fbe-7cdae76c6c34\" (UID: \"0e084ea3-7600-4ac5-9fbe-7cdae76c6c34\") "
Sep 17 11:12:14.290973 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.290961    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:12:14.291014 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.290976    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:12:14.291014 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.290986    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvz5q\" (UniqueName: \"kubernetes.io/projected/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-kube-api-access-vvz5q\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:12:14.291014 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.290994    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e58d8aae-4bd4-4d67-bd6b-0b27070383d6-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:12:14.291164 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.291053    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e084ea3-7600-4ac5-9fbe-7cdae76c6c34-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0e084ea3-7600-4ac5-9fbe-7cdae76c6c34" (UID: "0e084ea3-7600-4ac5-9fbe-7cdae76c6c34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:12:14.292760 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.292739    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e084ea3-7600-4ac5-9fbe-7cdae76c6c34-kube-api-access-2v55q" (OuterVolumeSpecName: "kube-api-access-2v55q") pod "0e084ea3-7600-4ac5-9fbe-7cdae76c6c34" (UID: "0e084ea3-7600-4ac5-9fbe-7cdae76c6c34"). InnerVolumeSpecName "kube-api-access-2v55q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:12:14.315317 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.315287    2562 generic.go:356] "Generic (PLEG): container finished" podID="0e084ea3-7600-4ac5-9fbe-7cdae76c6c34" containerID="84372e3ad95fc177d7b22565cf7bcccf36c83185553a71acfb2afa83a612f34d" exitCode=0
Sep 17 11:12:14.315479 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.315334    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v"
Sep 17 11:12:14.315479 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.315386    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v" event={"ID":"0e084ea3-7600-4ac5-9fbe-7cdae76c6c34","Type":"ContainerDied","Data":"84372e3ad95fc177d7b22565cf7bcccf36c83185553a71acfb2afa83a612f34d"}
Sep 17 11:12:14.315479 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.315435    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v" event={"ID":"0e084ea3-7600-4ac5-9fbe-7cdae76c6c34","Type":"ContainerDied","Data":"985e84ccfd1c8ac04bb1f9a1242ca12be76de0b3718454e113ed8f18b11d4ec7"}
Sep 17 11:12:14.315479 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.315468    2562 scope.go:117] "RemoveContainer" containerID="84372e3ad95fc177d7b22565cf7bcccf36c83185553a71acfb2afa83a612f34d"
Sep 17 11:12:14.316861 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.316841    2562 generic.go:356] "Generic (PLEG): container finished" podID="e58d8aae-4bd4-4d67-bd6b-0b27070383d6" containerID="d386835140de7eb7b34d1283460f47f72b047d8dd66d95298850e571503b7f72" exitCode=0
Sep 17 11:12:14.316957 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.316874    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp" event={"ID":"e58d8aae-4bd4-4d67-bd6b-0b27070383d6","Type":"ContainerDied","Data":"d386835140de7eb7b34d1283460f47f72b047d8dd66d95298850e571503b7f72"}
Sep 17 11:12:14.316957 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.316890    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"
Sep 17 11:12:14.316957 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.316930    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp" event={"ID":"e58d8aae-4bd4-4d67-bd6b-0b27070383d6","Type":"ContainerDied","Data":"080ffc2f03f964e71213932e03ab36264180c0f419e8cacc50373dac3575d62b"}
Sep 17 11:12:14.324866 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.324843    2562 scope.go:117] "RemoveContainer" containerID="84372e3ad95fc177d7b22565cf7bcccf36c83185553a71acfb2afa83a612f34d"
Sep 17 11:12:14.325190 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:12:14.325173    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84372e3ad95fc177d7b22565cf7bcccf36c83185553a71acfb2afa83a612f34d\": container with ID starting with 84372e3ad95fc177d7b22565cf7bcccf36c83185553a71acfb2afa83a612f34d not found: ID does not exist" containerID="84372e3ad95fc177d7b22565cf7bcccf36c83185553a71acfb2afa83a612f34d"
Sep 17 11:12:14.325252 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.325198    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84372e3ad95fc177d7b22565cf7bcccf36c83185553a71acfb2afa83a612f34d"} err="failed to get container status \"84372e3ad95fc177d7b22565cf7bcccf36c83185553a71acfb2afa83a612f34d\": rpc error: code = NotFound desc = could not find container \"84372e3ad95fc177d7b22565cf7bcccf36c83185553a71acfb2afa83a612f34d\": container with ID starting with 84372e3ad95fc177d7b22565cf7bcccf36c83185553a71acfb2afa83a612f34d not found: ID does not exist"
Sep 17 11:12:14.325252 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.325217    2562 scope.go:117] "RemoveContainer" containerID="d386835140de7eb7b34d1283460f47f72b047d8dd66d95298850e571503b7f72"
Sep 17 11:12:14.333706 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.333686    2562 scope.go:117] "RemoveContainer" containerID="d386835140de7eb7b34d1283460f47f72b047d8dd66d95298850e571503b7f72"
Sep 17 11:12:14.334010 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:12:14.333988    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d386835140de7eb7b34d1283460f47f72b047d8dd66d95298850e571503b7f72\": container with ID starting with d386835140de7eb7b34d1283460f47f72b047d8dd66d95298850e571503b7f72 not found: ID does not exist" containerID="d386835140de7eb7b34d1283460f47f72b047d8dd66d95298850e571503b7f72"
Sep 17 11:12:14.334062 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.334020    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d386835140de7eb7b34d1283460f47f72b047d8dd66d95298850e571503b7f72"} err="failed to get container status \"d386835140de7eb7b34d1283460f47f72b047d8dd66d95298850e571503b7f72\": rpc error: code = NotFound desc = could not find container \"d386835140de7eb7b34d1283460f47f72b047d8dd66d95298850e571503b7f72\": container with ID starting with d386835140de7eb7b34d1283460f47f72b047d8dd66d95298850e571503b7f72 not found: ID does not exist"
Sep 17 11:12:14.346754 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.346727    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"]
Sep 17 11:12:14.349859 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.349830    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15f9c946-75bb57bcc4-z2wjp"]
Sep 17 11:12:14.364684 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.364587    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v"]
Sep 17 11:12:14.371601 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.371562    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15f9c946-6c477bb5d5-8h47v"]
Sep 17 11:12:14.392465 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.392420    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e084ea3-7600-4ac5-9fbe-7cdae76c6c34-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:12:14.392465 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.392452    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v55q\" (UniqueName: \"kubernetes.io/projected/0e084ea3-7600-4ac5-9fbe-7cdae76c6c34-kube-api-access-2v55q\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:12:14.433951 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.433917    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:14.456328 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.456293    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e084ea3-7600-4ac5-9fbe-7cdae76c6c34" path="/var/lib/kubelet/pods/0e084ea3-7600-4ac5-9fbe-7cdae76c6c34/volumes"
Sep 17 11:12:14.456698 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.456685    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e58d8aae-4bd4-4d67-bd6b-0b27070383d6" path="/var/lib/kubelet/pods/e58d8aae-4bd4-4d67-bd6b-0b27070383d6/volumes"
Sep 17 11:12:14.506304 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.506270    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:14.585070 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:14.585016    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:15.284275 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:15.284244    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:15.616128 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:15.616033    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:15.842659 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:15.842630    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:16.559117 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:16.559085    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:16.988638 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:16.988556    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:17.516138 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.516107    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"]
Sep 17 11:12:17.516681 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.516657    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e58d8aae-4bd4-4d67-bd6b-0b27070383d6" containerName="maas-api"
Sep 17 11:12:17.516800 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.516683    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e58d8aae-4bd4-4d67-bd6b-0b27070383d6" containerName="maas-api"
Sep 17 11:12:17.516800 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.516698    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e084ea3-7600-4ac5-9fbe-7cdae76c6c34" containerName="payload-processing"
Sep 17 11:12:17.516800 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.516706    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e084ea3-7600-4ac5-9fbe-7cdae76c6c34" containerName="payload-processing"
Sep 17 11:12:17.516988 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.516827    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e58d8aae-4bd4-4d67-bd6b-0b27070383d6" containerName="maas-api"
Sep 17 11:12:17.516988 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.516848    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e084ea3-7600-4ac5-9fbe-7cdae76c6c34" containerName="payload-processing"
Sep 17 11:12:17.521589 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.521561    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:12:17.524643 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.524622    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-b8287f-serving-cert\""
Sep 17 11:12:17.533340 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.533318    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"]
Sep 17 11:12:17.553970 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.553935    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:17.616925 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.616891    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:17.617309 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.617089    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw\" (UID: \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:12:17.617309 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.617117    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8xmz\" (UniqueName: \"kubernetes.io/projected/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-kube-api-access-c8xmz\") pod \"maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw\" (UID: \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:12:17.617309 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.617138    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw\" (UID: \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:12:17.617309 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.617203    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw\" (UID: \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:12:17.718347 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.718305    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw\" (UID: \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:12:17.718536 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.718417    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw\" (UID: \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:12:17.718536 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.718435    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8xmz\" (UniqueName: \"kubernetes.io/projected/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-kube-api-access-c8xmz\") pod \"maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw\" (UID: \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:12:17.718536 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.718453    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw\" (UID: \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:12:17.718706 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:12:17.718578    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-serving-cert: secret "maas-api-e2e-shared-b-w2-b8287f-serving-cert" not found
Sep 17 11:12:17.718706 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:12:17.718681    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-maas-api-tls podName:d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41 nodeName:}" failed. No retries permitted until 2026-09-17 11:12:18.21865965 +0000 UTC m=+1132.316705213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-maas-api-tls") pod "maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw" (UID: "d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41") : secret "maas-api-e2e-shared-b-w2-b8287f-serving-cert" not found
Sep 17 11:12:17.718973 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.718954    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw\" (UID: \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:12:17.720975 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.720957    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw\" (UID: \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:12:17.730831 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:17.730798    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8xmz\" (UniqueName: \"kubernetes.io/projected/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-kube-api-access-c8xmz\") pod \"maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw\" (UID: \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:12:18.150546 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:18.150515    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:18.222940 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:18.222910    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw\" (UID: \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:12:18.225243 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:18.225223    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw\" (UID: \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:12:18.248861 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:18.248826    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:18.433008 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:18.432924    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:12:18.713028 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:18.713003    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"]
Sep 17 11:12:18.714975 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:12:18.714948    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd234cffa_5bc1_4fe2_8e62_4f8a9c55bd41.slice/crio-c6c74c3b9d3d272a419033a860fbd1dcc7612b24a9dd59d9b07f50c6b488fb46 WatchSource:0}: Error finding container c6c74c3b9d3d272a419033a860fbd1dcc7612b24a9dd59d9b07f50c6b488fb46: Status 404 returned error can't find the container with id c6c74c3b9d3d272a419033a860fbd1dcc7612b24a9dd59d9b07f50c6b488fb46
Sep 17 11:12:18.795009 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:18.794977    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:19.347384 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:19.347348    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw" event={"ID":"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41","Type":"ContainerStarted","Data":"14fe24e9f4c5341ea28f67ba0b53a07994087e10e969e145148e17ca40876a7c"}
Sep 17 11:12:19.347591 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:19.347425    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw" event={"ID":"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41","Type":"ContainerStarted","Data":"c6c74c3b9d3d272a419033a860fbd1dcc7612b24a9dd59d9b07f50c6b488fb46"}
Sep 17 11:12:19.347591 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:19.347507    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:12:19.383370 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:19.383311    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw" podStartSLOduration=1.8300442829999999 podStartE2EDuration="2.383294057s" totalImagesPullingTime="553.249774ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:12:19.38223406 +0000 UTC m=+1133.480279626" watchObservedRunningTime="2026-09-17 11:12:19.383294057 +0000 UTC m=+1133.481339622"
Sep 17 11:12:19.787298 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:19.787261    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:20.722998 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:20.722965    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:12:23.966731 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:23.966693    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:24.081974 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:24.081927    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:24.191052 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:24.191019    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:24.289506 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:24.289474    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:24.651952 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:24.651860    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:25.359210 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:25.359184    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:12:40.215077 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:40.215046    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:40.976975 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:40.976945    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:41.954970 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:41.954940    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:42.863989 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:42.863959    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:44.061530 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:44.061495    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:49.125737 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:49.125652    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:12:50.589208 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:50.589174    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:56.651970 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:56.651938    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:56.765782 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:56.765742    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:12:56.858242 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:12:56.858197    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:02.615885 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.615852    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"]
Sep 17 11:13:02.619490 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.619472    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.623968 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.623934    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-026f86\"/\"default-dockercfg-j5nx6\""
Sep 17 11:13:02.624114 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.623967    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-026f86\"/\"auth-test-model-026f86-kserve-self-signed-certs\""
Sep 17 11:13:02.624114 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.624041    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-026f86\"/\"openshift-service-ca.crt\""
Sep 17 11:13:02.624114 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.624082    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-026f86\"/\"kube-root-ca.crt\""
Sep 17 11:13:02.631690 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.631664    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"]
Sep 17 11:13:02.643834 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.643790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvs27\" (UniqueName: \"kubernetes.io/projected/ec21cb8a-a490-43bc-adb6-d4449b5cc177-kube-api-access-rvs27\") pod \"auth-test-model-026f86-kserve-6c945fb76c-7cvhz\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") " pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.643987 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.643853    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-model-cache\") pod \"auth-test-model-026f86-kserve-6c945fb76c-7cvhz\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") " pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.643987 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.643892    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-tmp-dir\") pod \"auth-test-model-026f86-kserve-6c945fb76c-7cvhz\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") " pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.643987 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.643913    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec21cb8a-a490-43bc-adb6-d4449b5cc177-tls-certs\") pod \"auth-test-model-026f86-kserve-6c945fb76c-7cvhz\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") " pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.643987 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.643968    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-home\") pod \"auth-test-model-026f86-kserve-6c945fb76c-7cvhz\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") " pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.644232 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.644047    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-dshm\") pod \"auth-test-model-026f86-kserve-6c945fb76c-7cvhz\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") " pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.653663 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.653629    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:02.744621 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.744587    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvs27\" (UniqueName: \"kubernetes.io/projected/ec21cb8a-a490-43bc-adb6-d4449b5cc177-kube-api-access-rvs27\") pod \"auth-test-model-026f86-kserve-6c945fb76c-7cvhz\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") " pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.744805 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.744637    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-model-cache\") pod \"auth-test-model-026f86-kserve-6c945fb76c-7cvhz\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") " pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.744805 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.744675    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-tmp-dir\") pod \"auth-test-model-026f86-kserve-6c945fb76c-7cvhz\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") " pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.744805 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.744692    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec21cb8a-a490-43bc-adb6-d4449b5cc177-tls-certs\") pod \"auth-test-model-026f86-kserve-6c945fb76c-7cvhz\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") " pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.744805 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.744707    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-home\") pod \"auth-test-model-026f86-kserve-6c945fb76c-7cvhz\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") " pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.745041 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.744945    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-dshm\") pod \"auth-test-model-026f86-kserve-6c945fb76c-7cvhz\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") " pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.745168 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.745135    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-model-cache\") pod \"auth-test-model-026f86-kserve-6c945fb76c-7cvhz\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") " pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.745168 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.745155    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-tmp-dir\") pod \"auth-test-model-026f86-kserve-6c945fb76c-7cvhz\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") " pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.745334 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.745190    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-home\") pod \"auth-test-model-026f86-kserve-6c945fb76c-7cvhz\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") " pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.747178 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.747149    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-dshm\") pod \"auth-test-model-026f86-kserve-6c945fb76c-7cvhz\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") " pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.747281 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.747213    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec21cb8a-a490-43bc-adb6-d4449b5cc177-tls-certs\") pod \"auth-test-model-026f86-kserve-6c945fb76c-7cvhz\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") " pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.753967 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.753942    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvs27\" (UniqueName: \"kubernetes.io/projected/ec21cb8a-a490-43bc-adb6-d4449b5cc177-kube-api-access-rvs27\") pod \"auth-test-model-026f86-kserve-6c945fb76c-7cvhz\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") " pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:02.929436 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:02.929344    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:03.202809 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:03.202783    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"]
Sep 17 11:13:03.204665 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:13:03.204637    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec21cb8a_a490_43bc_adb6_d4449b5cc177.slice/crio-e6228b521e85eab5a50d1abb5ad577ee2660d7e8308c647f134df472f876b96c WatchSource:0}: Error finding container e6228b521e85eab5a50d1abb5ad577ee2660d7e8308c647f134df472f876b96c: Status 404 returned error can't find the container with id e6228b521e85eab5a50d1abb5ad577ee2660d7e8308c647f134df472f876b96c
Sep 17 11:13:03.581649 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:03.581610    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz" event={"ID":"ec21cb8a-a490-43bc-adb6-d4449b5cc177","Type":"ContainerStarted","Data":"001e1b00f8e5c8f808874cf351e60a42436200a60d0819de99df9713b01bd707"}
Sep 17 11:13:03.581649 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:03.581654    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz" event={"ID":"ec21cb8a-a490-43bc-adb6-d4449b5cc177","Type":"ContainerStarted","Data":"e6228b521e85eab5a50d1abb5ad577ee2660d7e8308c647f134df472f876b96c"}
Sep 17 11:13:03.599689 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:03.599632    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz" podStartSLOduration=1.599616054 podStartE2EDuration="1.599616054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:13:03.598744683 +0000 UTC m=+1177.696790265" watchObservedRunningTime="2026-09-17 11:13:03.599616054 +0000 UTC m=+1177.697661672"
Sep 17 11:13:11.945772 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:11.945706    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:12.409639 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:12.409604    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:12.732281 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:12.732205    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:12.929741 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:12.929705    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:12.929741 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:12.929747    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:12.942606 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:12.942580    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:13.478705 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:13.478656    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:13.641782 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:13.641749    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:13:14.165266 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:14.165233    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:14.996440 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:14.996409    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:15.676361 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:15.676322    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:16.103889 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:16.103849    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:13:16.933037 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:16.933001    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"]
Sep 17 11:13:17.030522 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:17.030490    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:19.168224 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:19.168189    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:19.274877 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:19.274844    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:19.368219 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:19.368183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:23.807944 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:23.807914    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"]
Sep 17 11:13:23.811576 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:23.811557    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:23.814400 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:23.814378    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d7b97f\"/\"embedding-w0-d7b97f-kserve-self-signed-certs\""
Sep 17 11:13:23.821844 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:23.821810    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"]
Sep 17 11:13:23.837685 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:23.837645    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:23.941974 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:23.941936    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-tls-certs\") pod \"embedding-w0-d7b97f-kserve-68fbdb675-q9s9n\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") " pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:23.942173 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:23.941986    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-home\") pod \"embedding-w0-d7b97f-kserve-68fbdb675-q9s9n\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") " pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:23.942173 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:23.942097    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-dshm\") pod \"embedding-w0-d7b97f-kserve-68fbdb675-q9s9n\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") " pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:23.942173 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:23.942135    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq4lj\" (UniqueName: \"kubernetes.io/projected/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-kube-api-access-wq4lj\") pod \"embedding-w0-d7b97f-kserve-68fbdb675-q9s9n\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") " pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:23.942321 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:23.942231    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-model-cache\") pod \"embedding-w0-d7b97f-kserve-68fbdb675-q9s9n\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") " pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:23.942321 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:23.942272    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-tmp-dir\") pod \"embedding-w0-d7b97f-kserve-68fbdb675-q9s9n\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") " pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:24.043842 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:24.043774    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-home\") pod \"embedding-w0-d7b97f-kserve-68fbdb675-q9s9n\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") " pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:24.044040 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:24.043880    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-dshm\") pod \"embedding-w0-d7b97f-kserve-68fbdb675-q9s9n\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") " pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:24.044040 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:24.043908    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq4lj\" (UniqueName: \"kubernetes.io/projected/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-kube-api-access-wq4lj\") pod \"embedding-w0-d7b97f-kserve-68fbdb675-q9s9n\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") " pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:24.044040 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:24.043961    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-model-cache\") pod \"embedding-w0-d7b97f-kserve-68fbdb675-q9s9n\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") " pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:24.044040 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:24.044001    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-tmp-dir\") pod \"embedding-w0-d7b97f-kserve-68fbdb675-q9s9n\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") " pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:24.044270 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:24.044052    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-tls-certs\") pod \"embedding-w0-d7b97f-kserve-68fbdb675-q9s9n\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") " pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:24.044270 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:24.044197    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-home\") pod \"embedding-w0-d7b97f-kserve-68fbdb675-q9s9n\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") " pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:24.044380 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:24.044345    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-model-cache\") pod \"embedding-w0-d7b97f-kserve-68fbdb675-q9s9n\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") " pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:24.044380 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:24.044366    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-tmp-dir\") pod \"embedding-w0-d7b97f-kserve-68fbdb675-q9s9n\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") " pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:24.046250 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:24.046223    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-dshm\") pod \"embedding-w0-d7b97f-kserve-68fbdb675-q9s9n\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") " pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:24.046539 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:24.046520    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-tls-certs\") pod \"embedding-w0-d7b97f-kserve-68fbdb675-q9s9n\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") " pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:24.053044 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:24.053027    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq4lj\" (UniqueName: \"kubernetes.io/projected/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-kube-api-access-wq4lj\") pod \"embedding-w0-d7b97f-kserve-68fbdb675-q9s9n\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") " pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:24.121998 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:24.121904    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:24.399646 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:24.399620    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"]
Sep 17 11:13:24.402317 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:13:24.402289    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3df37b3_4dc3_4d68_b28e_beffc8257a9c.slice/crio-0309ad8c78a5e79715a7cc52e4aee983970b4edae83e70e92bfd8b5a39a27314 WatchSource:0}: Error finding container 0309ad8c78a5e79715a7cc52e4aee983970b4edae83e70e92bfd8b5a39a27314: Status 404 returned error can't find the container with id 0309ad8c78a5e79715a7cc52e4aee983970b4edae83e70e92bfd8b5a39a27314
Sep 17 11:13:24.688637 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:24.688547    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n" event={"ID":"f3df37b3-4dc3-4d68-b28e-beffc8257a9c","Type":"ContainerStarted","Data":"31da005ccbc3993f075daa4e2f581cd4b7bd59c916364a3a5f51ba8f4d07801b"}
Sep 17 11:13:24.688637 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:24.688590    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n" event={"ID":"f3df37b3-4dc3-4d68-b28e-beffc8257a9c","Type":"ContainerStarted","Data":"0309ad8c78a5e79715a7cc52e4aee983970b4edae83e70e92bfd8b5a39a27314"}
Sep 17 11:13:24.713013 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:24.712960    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n" podStartSLOduration=1.7129431990000001 podStartE2EDuration="1.712943199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:13:24.711866366 +0000 UTC m=+1198.809911948" watchObservedRunningTime="2026-09-17 11:13:24.712943199 +0000 UTC m=+1198.810988760"
Sep 17 11:13:26.452806 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:26.452774    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 11:13:26.454251 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:26.454227    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 11:13:27.669458 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:27.669427    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:27.763096 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:27.763063    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:27.849512 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:27.849478    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:27.968893 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:27.968772    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:28.046743 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:28.046710    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:28.374773 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:28.374739    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:28.665141 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:28.665060    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:29.393242 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:29.393211    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:29.896883 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:29.896847    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:30.356755 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:30.356714    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:30.680226 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:30.680132    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:31.952480 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:31.952418    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54" podUID="f19bbdec-e2e1-485a-aab3-c9fda81702a8" containerName="main" containerID="cri-o://8aaea066bb961edcf8803012bc4ba3960b0c5915aa656f4b53348f045b151377" gracePeriod=45
Sep 17 11:13:32.212155 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.212088    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:13:32.326662 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.326634    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-home\") pod \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") "
Sep 17 11:13:32.326868 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.326686    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-model-cache\") pod \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") "
Sep 17 11:13:32.326868 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.326770    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-dshm\") pod \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") "
Sep 17 11:13:32.326868 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.326834    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f19bbdec-e2e1-485a-aab3-c9fda81702a8-tls-certs\") pod \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") "
Sep 17 11:13:32.327062 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.326887    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph7s8\" (UniqueName: \"kubernetes.io/projected/f19bbdec-e2e1-485a-aab3-c9fda81702a8-kube-api-access-ph7s8\") pod \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") "
Sep 17 11:13:32.327062 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.326976    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-tmp-dir\") pod \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\" (UID: \"f19bbdec-e2e1-485a-aab3-c9fda81702a8\") "
Sep 17 11:13:32.327062 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.327000    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-home" (OuterVolumeSpecName: "home") pod "f19bbdec-e2e1-485a-aab3-c9fda81702a8" (UID: "f19bbdec-e2e1-485a-aab3-c9fda81702a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:13:32.327565 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.327308    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f19bbdec-e2e1-485a-aab3-c9fda81702a8" (UID: "f19bbdec-e2e1-485a-aab3-c9fda81702a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:13:32.327565 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.327367    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "f19bbdec-e2e1-485a-aab3-c9fda81702a8" (UID: "f19bbdec-e2e1-485a-aab3-c9fda81702a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:13:32.327565 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.327482    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:13:32.327565 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.327499    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:13:32.327565 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.327513    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:13:32.329329 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.329299    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-dshm" (OuterVolumeSpecName: "dshm") pod "f19bbdec-e2e1-485a-aab3-c9fda81702a8" (UID: "f19bbdec-e2e1-485a-aab3-c9fda81702a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:13:32.329548 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.329533    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f19bbdec-e2e1-485a-aab3-c9fda81702a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f19bbdec-e2e1-485a-aab3-c9fda81702a8" (UID: "f19bbdec-e2e1-485a-aab3-c9fda81702a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:13:32.329627 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.329609    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19bbdec-e2e1-485a-aab3-c9fda81702a8-kube-api-access-ph7s8" (OuterVolumeSpecName: "kube-api-access-ph7s8") pod "f19bbdec-e2e1-485a-aab3-c9fda81702a8" (UID: "f19bbdec-e2e1-485a-aab3-c9fda81702a8"). InnerVolumeSpecName "kube-api-access-ph7s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:13:32.428544 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.428504    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f19bbdec-e2e1-485a-aab3-c9fda81702a8-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:13:32.428544 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.428539    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f19bbdec-e2e1-485a-aab3-c9fda81702a8-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:13:32.428544 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.428550    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph7s8\" (UniqueName: \"kubernetes.io/projected/f19bbdec-e2e1-485a-aab3-c9fda81702a8-kube-api-access-ph7s8\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:13:32.731353 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.731268    2562 generic.go:356] "Generic (PLEG): container finished" podID="f19bbdec-e2e1-485a-aab3-c9fda81702a8" containerID="8aaea066bb961edcf8803012bc4ba3960b0c5915aa656f4b53348f045b151377" exitCode=0
Sep 17 11:13:32.731353 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.731306    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54" event={"ID":"f19bbdec-e2e1-485a-aab3-c9fda81702a8","Type":"ContainerDied","Data":"8aaea066bb961edcf8803012bc4ba3960b0c5915aa656f4b53348f045b151377"}
Sep 17 11:13:32.731353 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.731325    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"
Sep 17 11:13:32.731353 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.731343    2562 scope.go:117] "RemoveContainer" containerID="8aaea066bb961edcf8803012bc4ba3960b0c5915aa656f4b53348f045b151377"
Sep 17 11:13:32.731677 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.731333    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54" event={"ID":"f19bbdec-e2e1-485a-aab3-c9fda81702a8","Type":"ContainerDied","Data":"a3f98e4ff1845a6f47233c381f98fe5a0c3cf768954bec4ef852231d6c4f761d"}
Sep 17 11:13:32.740284 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.740265    2562 scope.go:117] "RemoveContainer" containerID="8aaea066bb961edcf8803012bc4ba3960b0c5915aa656f4b53348f045b151377"
Sep 17 11:13:32.740577 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:13:32.740556    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aaea066bb961edcf8803012bc4ba3960b0c5915aa656f4b53348f045b151377\": container with ID starting with 8aaea066bb961edcf8803012bc4ba3960b0c5915aa656f4b53348f045b151377 not found: ID does not exist" containerID="8aaea066bb961edcf8803012bc4ba3960b0c5915aa656f4b53348f045b151377"
Sep 17 11:13:32.740622 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.740585    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aaea066bb961edcf8803012bc4ba3960b0c5915aa656f4b53348f045b151377"} err="failed to get container status \"8aaea066bb961edcf8803012bc4ba3960b0c5915aa656f4b53348f045b151377\": rpc error: code = NotFound desc = could not find container \"8aaea066bb961edcf8803012bc4ba3960b0c5915aa656f4b53348f045b151377\": container with ID starting with 8aaea066bb961edcf8803012bc4ba3960b0c5915aa656f4b53348f045b151377 not found: ID does not exist"
Sep 17 11:13:32.750082 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.750061    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"]
Sep 17 11:13:32.759043 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:32.759024    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-766cc3/distinct-w3-766cc3-kserve-7777dd6596-7ss54"]
Sep 17 11:13:34.122587 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:34.122558    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:34.122968 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:34.122602    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:34.134642 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:34.134624    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:34.190409 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:34.190377    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:34.261962 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:34.261931    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:34.457666 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:34.457587    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f19bbdec-e2e1-485a-aab3-c9fda81702a8" path="/var/lib/kubelet/pods/f19bbdec-e2e1-485a-aab3-c9fda81702a8/volumes"
Sep 17 11:13:34.500313 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:34.500281    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:34.753498 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:34.753411    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:13:38.102000 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:38.101968    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:38.211711 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:38.211678    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:38.425416 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:38.425332    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:38.546227 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:38.546192    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:39.125670 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:39.125564    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:39.223606 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:39.223571    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:40.762480 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:40.762444    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:46.856287 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:46.856253    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:49.427009 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:49.426975    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:52.914801 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:52.914770    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:13:53.919523 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:53.919494    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:54.022500 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:54.022466    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:54.580160 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:54.580120    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:54.687032 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:54.687003    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:54.831287 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:54.831202    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:55.039886 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:55.039839    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:55.161591 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:55.161481    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:55.343387 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:55.343345    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:57.583257 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:57.583196    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:58.083866 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:58.083835    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:58.195546 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:58.195513    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:58.534199 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:58.534176    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:13:59.369393 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:13:59.369356    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:00.147135 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.147099    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"]
Sep 17 11:14:00.147803 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.147782    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f19bbdec-e2e1-485a-aab3-c9fda81702a8" containerName="main"
Sep 17 11:14:00.147892 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.147807    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f19bbdec-e2e1-485a-aab3-c9fda81702a8" containerName="main"
Sep 17 11:14:00.147959 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.147946    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f19bbdec-e2e1-485a-aab3-c9fda81702a8" containerName="main"
Sep 17 11:14:00.151648 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.151626    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:00.156589 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.156569    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d078f791-serving-cert\""
Sep 17 11:14:00.163424 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.163401    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"]
Sep 17 11:14:00.196858 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.196828    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:00.200112 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.200092    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-openshift-service-ca\") pod \"maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs\" (UID: \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:00.200180 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.200127    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-maas-api-tls\") pod \"maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs\" (UID: \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:00.200240 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.200222    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pj7x\" (UniqueName: \"kubernetes.io/projected/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-kube-api-access-5pj7x\") pod \"maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs\" (UID: \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:00.200277 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.200252    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-metrics-certs\") pod \"maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs\" (UID: \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:00.300486 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.300458    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:00.300759 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.300738    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-openshift-service-ca\") pod \"maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs\" (UID: \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:00.300845 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.300776    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-maas-api-tls\") pod \"maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs\" (UID: \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:00.300893 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.300870    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pj7x\" (UniqueName: \"kubernetes.io/projected/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-kube-api-access-5pj7x\") pod \"maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs\" (UID: \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:00.300937 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.300896    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-metrics-certs\") pod \"maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs\" (UID: \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:00.300981 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:14:00.300963    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-serving-cert: secret "maas-api-e2e-mt-d078f791-serving-cert" not found
Sep 17 11:14:00.301049 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:14:00.301039    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-maas-api-tls podName:138f92a3-bb3b-4d74-a0af-ddca94dd1f98 nodeName:}" failed. No retries permitted until 2026-09-17 11:14:00.801019053 +0000 UTC m=+1234.899064615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-maas-api-tls") pod "maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs" (UID: "138f92a3-bb3b-4d74-a0af-ddca94dd1f98") : secret "maas-api-e2e-mt-d078f791-serving-cert" not found
Sep 17 11:14:00.301494 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.301475    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-openshift-service-ca\") pod \"maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs\" (UID: \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:00.303317 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.303300    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-metrics-certs\") pod \"maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs\" (UID: \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:00.313169 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.313149    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pj7x\" (UniqueName: \"kubernetes.io/projected/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-kube-api-access-5pj7x\") pod \"maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs\" (UID: \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:00.751856 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.751798    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:00.806510 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.806472    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-maas-api-tls\") pod \"maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs\" (UID: \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:00.809020 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.808996    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-maas-api-tls\") pod \"maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs\" (UID: \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:00.888067 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:00.888031    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:01.003371 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:01.003285    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:01.062975 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:01.062937    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:01.344278 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:01.344244    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"]
Sep 17 11:14:01.347973 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:14:01.347944    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138f92a3_bb3b_4d74_a0af_ddca94dd1f98.slice/crio-477db0a6faa5228eddf322475e8f0021b1212fa40f5cf8594ff9bf7b8312c9cd WatchSource:0}: Error finding container 477db0a6faa5228eddf322475e8f0021b1212fa40f5cf8594ff9bf7b8312c9cd: Status 404 returned error can't find the container with id 477db0a6faa5228eddf322475e8f0021b1212fa40f5cf8594ff9bf7b8312c9cd
Sep 17 11:14:01.881362 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:01.881337    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs" event={"ID":"138f92a3-bb3b-4d74-a0af-ddca94dd1f98","Type":"ContainerStarted","Data":"477db0a6faa5228eddf322475e8f0021b1212fa40f5cf8594ff9bf7b8312c9cd"}
Sep 17 11:14:02.888145 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:02.888106    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs" event={"ID":"138f92a3-bb3b-4d74-a0af-ddca94dd1f98","Type":"ContainerStarted","Data":"28ab5d74eb85d1a01ae4cd5520a3db3a6e105b273a33c8fc47b2b72ea19f5b93"}
Sep 17 11:14:02.888555 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:02.888227    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:02.923455 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:02.923396    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs" podStartSLOduration=2.396451733 podStartE2EDuration="2.923377634s" totalImagesPullingTime="526.925901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:14:02.919987291 +0000 UTC m=+1237.018032856" watchObservedRunningTime="2026-09-17 11:14:02.923377634 +0000 UTC m=+1237.021423241"
Sep 17 11:14:08.899715 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:08.899687    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:13.250308 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:13.250265    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:14:21.088203 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:21.088128    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:21.466793 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:21.466710    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:22.666183 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:22.666147    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:23.616545 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:23.616510    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:24.009429 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:24.009340    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:24.570120 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:24.570083    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:25.461258 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:25.461221    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:26.042175 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:26.042139    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"]
Sep 17 11:14:26.054205 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:26.054174    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"]
Sep 17 11:14:26.171882 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:26.171847    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:26.277609 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:26.277574    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:27.249469 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:27.249440    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:28.086133 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:28.086095    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:28.746991 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:28.746962    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:28.842862 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:28.842801    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:29.234413 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:29.234378    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:31.354740 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:31.354709    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:35.164218 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:35.164185    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:35.269236 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:35.269198    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:35.860829 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:35.860776    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:36.862451 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:36.862414    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:36.921873 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:36.921840    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs"]
Sep 17 11:14:36.922105 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:36.922084    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs" podUID="7b686a84-7754-4679-b61f-1f927c4fc965" containerName="payload-pre-processing" containerID="cri-o://6c2e580295d391cd727eda46e41b93663fc1ba873a12702c9ca66046955265f5" gracePeriod=30
Sep 17 11:14:36.963707 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:36.963673    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:37.063481 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:37.063442    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:37.064596 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:37.064571    2562 generic.go:356] "Generic (PLEG): container finished" podID="7b686a84-7754-4679-b61f-1f927c4fc965" containerID="6c2e580295d391cd727eda46e41b93663fc1ba873a12702c9ca66046955265f5" exitCode=0
Sep 17 11:14:37.064696 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:37.064645    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs" event={"ID":"7b686a84-7754-4679-b61f-1f927c4fc965","Type":"ContainerDied","Data":"6c2e580295d391cd727eda46e41b93663fc1ba873a12702c9ca66046955265f5"}
Sep 17 11:14:37.164059 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:37.163972    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:37.181339 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:37.181315    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs"
Sep 17 11:14:37.260273 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:37.260240    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx448\" (UniqueName: \"kubernetes.io/projected/7b686a84-7754-4679-b61f-1f927c4fc965-kube-api-access-fx448\") pod \"7b686a84-7754-4679-b61f-1f927c4fc965\" (UID: \"7b686a84-7754-4679-b61f-1f927c4fc965\") "
Sep 17 11:14:37.260457 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:37.260324    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b686a84-7754-4679-b61f-1f927c4fc965-plugins-config-volume\") pod \"7b686a84-7754-4679-b61f-1f927c4fc965\" (UID: \"7b686a84-7754-4679-b61f-1f927c4fc965\") "
Sep 17 11:14:37.260732 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:37.260708    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b686a84-7754-4679-b61f-1f927c4fc965-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7b686a84-7754-4679-b61f-1f927c4fc965" (UID: "7b686a84-7754-4679-b61f-1f927c4fc965"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:14:37.262394 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:37.262376    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b686a84-7754-4679-b61f-1f927c4fc965-kube-api-access-fx448" (OuterVolumeSpecName: "kube-api-access-fx448") pod "7b686a84-7754-4679-b61f-1f927c4fc965" (UID: "7b686a84-7754-4679-b61f-1f927c4fc965"). InnerVolumeSpecName "kube-api-access-fx448". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:14:37.264520 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:37.264498    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:37.361498 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:37.361459    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx448\" (UniqueName: \"kubernetes.io/projected/7b686a84-7754-4679-b61f-1f927c4fc965-kube-api-access-fx448\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:14:37.361498 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:37.361491    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b686a84-7754-4679-b61f-1f927c4fc965-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:14:37.496516 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:37.496439    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:38.051205 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:38.051170    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:38.071077 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:38.071044    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs" event={"ID":"7b686a84-7754-4679-b61f-1f927c4fc965","Type":"ContainerDied","Data":"87a9930d5a1d512368f402d18ac132adc0186b4e8f9356cad45efda8c777fa1a"}
Sep 17 11:14:38.071268 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:38.071057    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs"
Sep 17 11:14:38.071268 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:38.071146    2562 scope.go:117] "RemoveContainer" containerID="6c2e580295d391cd727eda46e41b93663fc1ba873a12702c9ca66046955265f5"
Sep 17 11:14:38.093970 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:38.093938    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs"]
Sep 17 11:14:38.099255 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:38.099232    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d7b97f-5c67fccd8b-97nxs"]
Sep 17 11:14:38.456553 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:38.456476    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b686a84-7754-4679-b61f-1f927c4fc965" path="/var/lib/kubelet/pods/7b686a84-7754-4679-b61f-1f927c4fc965/volumes"
Sep 17 11:14:38.731984 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:38.731908    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:14:38.952965 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:38.952930    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:39.295673 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:39.295641    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:39.836368 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:39.836335    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:41.061901 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.061854    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n" podUID="f3df37b3-4dc3-4d68-b28e-beffc8257a9c" containerName="main" containerID="cri-o://31da005ccbc3993f075daa4e2f581cd4b7bd59c916364a3a5f51ba8f4d07801b" gracePeriod=45
Sep 17 11:14:41.073267 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.073235    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh" podUID="dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1" containerName="main" containerID="cri-o://172cf5f0a0add42d5cb95b5514e3c8cf37b15192b4e456bec17e1fa58e60dac4" gracePeriod=45
Sep 17 11:14:41.343406 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.343374    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:14:41.347301 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.347274    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:14:41.400615 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.400586    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-tls-certs\") pod \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") "
Sep 17 11:14:41.400795 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.400643    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-home\") pod \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") "
Sep 17 11:14:41.400795 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.400678    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-dshm\") pod \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") "
Sep 17 11:14:41.400795 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.400705    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-model-cache\") pod \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") "
Sep 17 11:14:41.400795 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.400750    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-tmp-dir\") pod \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") "
Sep 17 11:14:41.400795 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.400785    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-dshm\") pod \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") "
Sep 17 11:14:41.401104 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.400810    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq4lj\" (UniqueName: \"kubernetes.io/projected/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-kube-api-access-wq4lj\") pod \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") "
Sep 17 11:14:41.401104 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.400886    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kx9s\" (UniqueName: \"kubernetes.io/projected/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-kube-api-access-9kx9s\") pod \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") "
Sep 17 11:14:41.401104 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.400926    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-model-cache\") pod \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") "
Sep 17 11:14:41.401104 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.400935    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-home" (OuterVolumeSpecName: "home") pod "dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1" (UID: "dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:14:41.401104 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.400950    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-tls-certs\") pod \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") "
Sep 17 11:14:41.401104 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.401007    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-home\") pod \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\" (UID: \"f3df37b3-4dc3-4d68-b28e-beffc8257a9c\") "
Sep 17 11:14:41.401104 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.401064    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-tmp-dir\") pod \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\" (UID: \"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1\") "
Sep 17 11:14:41.401104 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.401101    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3df37b3-4dc3-4d68-b28e-beffc8257a9c" (UID: "f3df37b3-4dc3-4d68-b28e-beffc8257a9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:14:41.401500 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.401239    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3df37b3-4dc3-4d68-b28e-beffc8257a9c" (UID: "f3df37b3-4dc3-4d68-b28e-beffc8257a9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:14:41.401500 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.401290    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-model-cache" (OuterVolumeSpecName: "model-cache") pod "dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1" (UID: "dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:14:41.401500 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.401440    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:14:41.401500 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.401457    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:14:41.401500 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.401473    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:14:41.401500 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.401488    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:14:41.401955 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.401918    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1" (UID: "dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:14:41.402075 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.401980    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-home" (OuterVolumeSpecName: "home") pod "f3df37b3-4dc3-4d68-b28e-beffc8257a9c" (UID: "f3df37b3-4dc3-4d68-b28e-beffc8257a9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:14:41.404651 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.404617    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1" (UID: "dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:14:41.404651 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.404632    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-kube-api-access-wq4lj" (OuterVolumeSpecName: "kube-api-access-wq4lj") pod "f3df37b3-4dc3-4d68-b28e-beffc8257a9c" (UID: "f3df37b3-4dc3-4d68-b28e-beffc8257a9c"). InnerVolumeSpecName "kube-api-access-wq4lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:14:41.405073 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.405041    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-dshm" (OuterVolumeSpecName: "dshm") pod "dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1" (UID: "dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:14:41.405261 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.405242    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3df37b3-4dc3-4d68-b28e-beffc8257a9c" (UID: "f3df37b3-4dc3-4d68-b28e-beffc8257a9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:14:41.405739 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.405701    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-kube-api-access-9kx9s" (OuterVolumeSpecName: "kube-api-access-9kx9s") pod "dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1" (UID: "dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1"). InnerVolumeSpecName "kube-api-access-9kx9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:14:41.406708 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.406683    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-dshm" (OuterVolumeSpecName: "dshm") pod "f3df37b3-4dc3-4d68-b28e-beffc8257a9c" (UID: "f3df37b3-4dc3-4d68-b28e-beffc8257a9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:14:41.502201 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.502163    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:14:41.502201 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.502195    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:14:41.502201 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.502206    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:14:41.502447 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.502218    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq4lj\" (UniqueName: \"kubernetes.io/projected/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-kube-api-access-wq4lj\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:14:41.502447 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.502231    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kx9s\" (UniqueName: \"kubernetes.io/projected/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-kube-api-access-9kx9s\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:14:41.502447 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.502243    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:14:41.502447 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.502254    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3df37b3-4dc3-4d68-b28e-beffc8257a9c-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:14:41.502447 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:41.502263    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:14:42.094044 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.094011    2562 generic.go:356] "Generic (PLEG): container finished" podID="f3df37b3-4dc3-4d68-b28e-beffc8257a9c" containerID="31da005ccbc3993f075daa4e2f581cd4b7bd59c916364a3a5f51ba8f4d07801b" exitCode=0
Sep 17 11:14:42.094534 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.094075    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"
Sep 17 11:14:42.094534 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.094104    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n" event={"ID":"f3df37b3-4dc3-4d68-b28e-beffc8257a9c","Type":"ContainerDied","Data":"31da005ccbc3993f075daa4e2f581cd4b7bd59c916364a3a5f51ba8f4d07801b"}
Sep 17 11:14:42.094534 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.094143    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n" event={"ID":"f3df37b3-4dc3-4d68-b28e-beffc8257a9c","Type":"ContainerDied","Data":"0309ad8c78a5e79715a7cc52e4aee983970b4edae83e70e92bfd8b5a39a27314"}
Sep 17 11:14:42.094534 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.094162    2562 scope.go:117] "RemoveContainer" containerID="31da005ccbc3993f075daa4e2f581cd4b7bd59c916364a3a5f51ba8f4d07801b"
Sep 17 11:14:42.095612 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.095587    2562 generic.go:356] "Generic (PLEG): container finished" podID="dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1" containerID="172cf5f0a0add42d5cb95b5514e3c8cf37b15192b4e456bec17e1fa58e60dac4" exitCode=0
Sep 17 11:14:42.095743 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.095633    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"
Sep 17 11:14:42.095743 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.095675    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh" event={"ID":"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1","Type":"ContainerDied","Data":"172cf5f0a0add42d5cb95b5514e3c8cf37b15192b4e456bec17e1fa58e60dac4"}
Sep 17 11:14:42.095743 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.095720    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh" event={"ID":"dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1","Type":"ContainerDied","Data":"a7fab5169de79aab5d5d47d4b1beecf7845b5e8f893c81608799acf444b307df"}
Sep 17 11:14:42.103842 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.103807    2562 scope.go:117] "RemoveContainer" containerID="31da005ccbc3993f075daa4e2f581cd4b7bd59c916364a3a5f51ba8f4d07801b"
Sep 17 11:14:42.104159 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:14:42.104140    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31da005ccbc3993f075daa4e2f581cd4b7bd59c916364a3a5f51ba8f4d07801b\": container with ID starting with 31da005ccbc3993f075daa4e2f581cd4b7bd59c916364a3a5f51ba8f4d07801b not found: ID does not exist" containerID="31da005ccbc3993f075daa4e2f581cd4b7bd59c916364a3a5f51ba8f4d07801b"
Sep 17 11:14:42.104218 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.104170    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31da005ccbc3993f075daa4e2f581cd4b7bd59c916364a3a5f51ba8f4d07801b"} err="failed to get container status \"31da005ccbc3993f075daa4e2f581cd4b7bd59c916364a3a5f51ba8f4d07801b\": rpc error: code = NotFound desc = could not find container \"31da005ccbc3993f075daa4e2f581cd4b7bd59c916364a3a5f51ba8f4d07801b\": container with ID starting with 31da005ccbc3993f075daa4e2f581cd4b7bd59c916364a3a5f51ba8f4d07801b not found: ID does not exist"
Sep 17 11:14:42.104218 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.104187    2562 scope.go:117] "RemoveContainer" containerID="172cf5f0a0add42d5cb95b5514e3c8cf37b15192b4e456bec17e1fa58e60dac4"
Sep 17 11:14:42.112449 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.112430    2562 scope.go:117] "RemoveContainer" containerID="172cf5f0a0add42d5cb95b5514e3c8cf37b15192b4e456bec17e1fa58e60dac4"
Sep 17 11:14:42.112739 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:14:42.112719    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172cf5f0a0add42d5cb95b5514e3c8cf37b15192b4e456bec17e1fa58e60dac4\": container with ID starting with 172cf5f0a0add42d5cb95b5514e3c8cf37b15192b4e456bec17e1fa58e60dac4 not found: ID does not exist" containerID="172cf5f0a0add42d5cb95b5514e3c8cf37b15192b4e456bec17e1fa58e60dac4"
Sep 17 11:14:42.112793 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.112748    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172cf5f0a0add42d5cb95b5514e3c8cf37b15192b4e456bec17e1fa58e60dac4"} err="failed to get container status \"172cf5f0a0add42d5cb95b5514e3c8cf37b15192b4e456bec17e1fa58e60dac4\": rpc error: code = NotFound desc = could not find container \"172cf5f0a0add42d5cb95b5514e3c8cf37b15192b4e456bec17e1fa58e60dac4\": container with ID starting with 172cf5f0a0add42d5cb95b5514e3c8cf37b15192b4e456bec17e1fa58e60dac4 not found: ID does not exist"
Sep 17 11:14:42.121656 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.121626    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"]
Sep 17 11:14:42.126074 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.126040    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d7b97f/embedding-w0-d7b97f-kserve-68fbdb675-q9s9n"]
Sep 17 11:14:42.139394 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.139363    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"]
Sep 17 11:14:42.143057 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.143026    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d7b97f/premium-w0-d7b97f-kserve-5c8b6996d7-mqnlh"]
Sep 17 11:14:42.456708 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.456626    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1" path="/var/lib/kubelet/pods/dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1/volumes"
Sep 17 11:14:42.457061 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:42.457047    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3df37b3-4dc3-4d68-b28e-beffc8257a9c" path="/var/lib/kubelet/pods/f3df37b3-4dc3-4d68-b28e-beffc8257a9c/volumes"
Sep 17 11:14:43.788510 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:43.788476    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:43.888884 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:43.888848    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:44.198988 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:44.198904    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:45.767331 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:45.767294    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:46.425300 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.425263    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"]
Sep 17 11:14:46.425530 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.425505    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs" podUID="138f92a3-bb3b-4d74-a0af-ddca94dd1f98" containerName="maas-api" containerID="cri-o://28ab5d74eb85d1a01ae4cd5520a3db3a6e105b273a33c8fc47b2b72ea19f5b93" gracePeriod=30
Sep 17 11:14:46.453651 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.453608    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:46.578385 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.578351    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:46.682740 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.682681    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:46.752484 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.752451    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pj7x\" (UniqueName: \"kubernetes.io/projected/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-kube-api-access-5pj7x\") pod \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\" (UID: \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\") "
Sep 17 11:14:46.752661 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.752525    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-openshift-service-ca\") pod \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\" (UID: \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\") "
Sep 17 11:14:46.752719 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.752665    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-maas-api-tls\") pod \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\" (UID: \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\") "
Sep 17 11:14:46.752767 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.752738    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-metrics-certs\") pod \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\" (UID: \"138f92a3-bb3b-4d74-a0af-ddca94dd1f98\") "
Sep 17 11:14:46.753068 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.752926    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "138f92a3-bb3b-4d74-a0af-ddca94dd1f98" (UID: "138f92a3-bb3b-4d74-a0af-ddca94dd1f98"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:14:46.753171 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.753143    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:14:46.755185 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.755130    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-kube-api-access-5pj7x" (OuterVolumeSpecName: "kube-api-access-5pj7x") pod "138f92a3-bb3b-4d74-a0af-ddca94dd1f98" (UID: "138f92a3-bb3b-4d74-a0af-ddca94dd1f98"). InnerVolumeSpecName "kube-api-access-5pj7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:14:46.755396 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.755371    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "138f92a3-bb3b-4d74-a0af-ddca94dd1f98" (UID: "138f92a3-bb3b-4d74-a0af-ddca94dd1f98"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:14:46.755551 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.755525    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "138f92a3-bb3b-4d74-a0af-ddca94dd1f98" (UID: "138f92a3-bb3b-4d74-a0af-ddca94dd1f98"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:14:46.769679 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.769656    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:46.840029 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.839996    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:46.854292 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.854262    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pj7x\" (UniqueName: \"kubernetes.io/projected/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-kube-api-access-5pj7x\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:14:46.854292 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.854289    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:14:46.854292 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:46.854300    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/138f92a3-bb3b-4d74-a0af-ddca94dd1f98-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:14:47.123692 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:47.123658    2562 generic.go:356] "Generic (PLEG): container finished" podID="138f92a3-bb3b-4d74-a0af-ddca94dd1f98" containerID="28ab5d74eb85d1a01ae4cd5520a3db3a6e105b273a33c8fc47b2b72ea19f5b93" exitCode=0
Sep 17 11:14:47.123870 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:47.123698    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"
Sep 17 11:14:47.123870 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:47.123746    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs" event={"ID":"138f92a3-bb3b-4d74-a0af-ddca94dd1f98","Type":"ContainerDied","Data":"28ab5d74eb85d1a01ae4cd5520a3db3a6e105b273a33c8fc47b2b72ea19f5b93"}
Sep 17 11:14:47.123870 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:47.123791    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs" event={"ID":"138f92a3-bb3b-4d74-a0af-ddca94dd1f98","Type":"ContainerDied","Data":"477db0a6faa5228eddf322475e8f0021b1212fa40f5cf8594ff9bf7b8312c9cd"}
Sep 17 11:14:47.123870 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:47.123839    2562 scope.go:117] "RemoveContainer" containerID="28ab5d74eb85d1a01ae4cd5520a3db3a6e105b273a33c8fc47b2b72ea19f5b93"
Sep 17 11:14:47.133094 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:47.133072    2562 scope.go:117] "RemoveContainer" containerID="28ab5d74eb85d1a01ae4cd5520a3db3a6e105b273a33c8fc47b2b72ea19f5b93"
Sep 17 11:14:47.133347 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:14:47.133330    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ab5d74eb85d1a01ae4cd5520a3db3a6e105b273a33c8fc47b2b72ea19f5b93\": container with ID starting with 28ab5d74eb85d1a01ae4cd5520a3db3a6e105b273a33c8fc47b2b72ea19f5b93 not found: ID does not exist" containerID="28ab5d74eb85d1a01ae4cd5520a3db3a6e105b273a33c8fc47b2b72ea19f5b93"
Sep 17 11:14:47.133392 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:47.133355    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ab5d74eb85d1a01ae4cd5520a3db3a6e105b273a33c8fc47b2b72ea19f5b93"} err="failed to get container status \"28ab5d74eb85d1a01ae4cd5520a3db3a6e105b273a33c8fc47b2b72ea19f5b93\": rpc error: code = NotFound desc = could not find container \"28ab5d74eb85d1a01ae4cd5520a3db3a6e105b273a33c8fc47b2b72ea19f5b93\": container with ID starting with 28ab5d74eb85d1a01ae4cd5520a3db3a6e105b273a33c8fc47b2b72ea19f5b93 not found: ID does not exist"
Sep 17 11:14:47.171002 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:47.170971    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"]
Sep 17 11:14:47.181932 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:47.181907    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d078f791-7d546fb59c-nr6vs"]
Sep 17 11:14:47.315622 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:47.315579    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:47.495373 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:47.495269    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:48.455353 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:48.455321    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="138f92a3-bb3b-4d74-a0af-ddca94dd1f98" path="/var/lib/kubelet/pods/138f92a3-bb3b-4d74-a0af-ddca94dd1f98/volumes"
Sep 17 11:14:48.644471 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:48.644440    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:53.443470 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:53.443437    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:53.542216 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:53.542187    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:53.723572 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:53.723485    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:54.052522 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:54.052494    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:54.144268 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:54.144231    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:54.259068 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:54.259037    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:14:59.918460 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:14:59.918423    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:00.718033 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:00.717998    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:00.882147 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:00.882111    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:01.135127 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:01.135092    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:01.264371 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:01.264335    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:02.426542 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:02.426502    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:15:07.069034 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:07.069000    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:07.166176 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:07.166142    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:07.982552 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:07.982519    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:08.338298 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:08.338260    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:08.871270 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:08.871233    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:11.158778 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:11.158746    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:11.259652 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:11.259618    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:11.391555 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:11.391520    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:11.483790 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:11.483683    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:11.585408 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:11.585375    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:12.682074 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:12.682046    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:12.781757 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:12.781725    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:12.958584 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:12.958502    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:13.194151 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:13.194111    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:13.286414 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:13.286381    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:13.425430 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:13.425400    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:15.116289 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:15.116255    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:18.221730 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:18.221698    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:15:20.923947 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:20.923912    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:26.116035 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:26.116004    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:26.216523 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:26.216485    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:32.011881 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:32.011845    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:15:35.515590 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:35.515556    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:36.067527 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:36.067494    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:36.809739 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:36.809709    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:37.017569 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:37.017530    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:37.248555 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:37.248468    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:37.454212 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:37.454167    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:40.095609 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:40.095572    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:40.202213 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:40.202182    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:40.520224 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:40.520129    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:41.662553 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:41.662525    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:42.555287 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.555256    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"]
Sep 17 11:15:42.555769 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.555753    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b686a84-7754-4679-b61f-1f927c4fc965" containerName="payload-pre-processing"
Sep 17 11:15:42.555832 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.555771    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b686a84-7754-4679-b61f-1f927c4fc965" containerName="payload-pre-processing"
Sep 17 11:15:42.555832 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.555780    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="138f92a3-bb3b-4d74-a0af-ddca94dd1f98" containerName="maas-api"
Sep 17 11:15:42.555832 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.555786    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="138f92a3-bb3b-4d74-a0af-ddca94dd1f98" containerName="maas-api"
Sep 17 11:15:42.555832 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.555802    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1" containerName="main"
Sep 17 11:15:42.555832 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.555808    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1" containerName="main"
Sep 17 11:15:42.555993 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.555845    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3df37b3-4dc3-4d68-b28e-beffc8257a9c" containerName="main"
Sep 17 11:15:42.555993 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.555854    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3df37b3-4dc3-4d68-b28e-beffc8257a9c" containerName="main"
Sep 17 11:15:42.555993 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.555930    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd4df3cf-5c57-4753-a2e4-a4bbb3b22fd1" containerName="main"
Sep 17 11:15:42.555993 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.555937    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b686a84-7754-4679-b61f-1f927c4fc965" containerName="payload-pre-processing"
Sep 17 11:15:42.555993 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.555943    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="138f92a3-bb3b-4d74-a0af-ddca94dd1f98" containerName="maas-api"
Sep 17 11:15:42.555993 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.555951    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3df37b3-4dc3-4d68-b28e-beffc8257a9c" containerName="main"
Sep 17 11:15:42.558703 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.558684    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:15:42.562158 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.562136    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-947f83-serving-cert\""
Sep 17 11:15:42.572137 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.572075    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"]
Sep 17 11:15:42.599524 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.599488    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:42.671385 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.671353    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:42.676665 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.676640    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2lj6\" (UniqueName: \"kubernetes.io/projected/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-kube-api-access-z2lj6\") pod \"maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r\" (UID: \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:15:42.676735 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.676701    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r\" (UID: \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:15:42.676773 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.676763    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r\" (UID: \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:15:42.676841 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.676826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r\" (UID: \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:15:42.778187 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.778147    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r\" (UID: \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:15:42.778369 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.778197    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r\" (UID: \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:15:42.778369 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.778347    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2lj6\" (UniqueName: \"kubernetes.io/projected/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-kube-api-access-z2lj6\") pod \"maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r\" (UID: \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:15:42.778454 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.778426    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r\" (UID: \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:15:42.778602 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:15:42.778577    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-serving-cert: secret "maas-api-e2e-shared-b-w1-947f83-serving-cert" not found
Sep 17 11:15:42.778670 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:15:42.778659    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-maas-api-tls podName:ac0bdb7b-c601-4d7a-84a9-936f1e228b34 nodeName:}" failed. No retries permitted until 2026-09-17 11:15:43.27863578 +0000 UTC m=+1337.376681336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-maas-api-tls") pod "maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r" (UID: "ac0bdb7b-c601-4d7a-84a9-936f1e228b34") : secret "maas-api-e2e-shared-b-w1-947f83-serving-cert" not found
Sep 17 11:15:42.778985 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.778966    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r\" (UID: \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:15:42.780697 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.780675    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r\" (UID: \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:15:42.787453 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.787430    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2lj6\" (UniqueName: \"kubernetes.io/projected/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-kube-api-access-z2lj6\") pod \"maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r\" (UID: \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:15:42.892172 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:42.892090    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:43.283451 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:43.283418    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r\" (UID: \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:15:43.285912 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:43.285889    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r\" (UID: \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:15:43.367889 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:43.367860    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:43.469317 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:43.469275    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:15:43.486830 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:43.486785    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:43.752328 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:43.752255    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"]
Sep 17 11:15:43.755759 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:15:43.755734    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0bdb7b_c601_4d7a_84a9_936f1e228b34.slice/crio-491a13668950c7192a7b92878f5f6751dd7102a36299b8ee0226ab3f7339c6e5 WatchSource:0}: Error finding container 491a13668950c7192a7b92878f5f6751dd7102a36299b8ee0226ab3f7339c6e5: Status 404 returned error can't find the container with id 491a13668950c7192a7b92878f5f6751dd7102a36299b8ee0226ab3f7339c6e5
Sep 17 11:15:44.162913 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:44.162857    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:44.411737 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:44.411704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r" event={"ID":"ac0bdb7b-c601-4d7a-84a9-936f1e228b34","Type":"ContainerStarted","Data":"f03de38c42286584147a6927b957768b0e1fe6e2ba2ce0fba53538dcbe5a5e3f"}
Sep 17 11:15:44.411893 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:44.411753    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r" event={"ID":"ac0bdb7b-c601-4d7a-84a9-936f1e228b34","Type":"ContainerStarted","Data":"491a13668950c7192a7b92878f5f6751dd7102a36299b8ee0226ab3f7339c6e5"}
Sep 17 11:15:44.411893 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:44.411856    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:15:44.438309 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:44.438212    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r" podStartSLOduration=2.011313901 podStartE2EDuration="2.438195503s" totalImagesPullingTime="426.881602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:15:44.436130902 +0000 UTC m=+1338.534176473" watchObservedRunningTime="2026-09-17 11:15:44.438195503 +0000 UTC m=+1338.536241069"
Sep 17 11:15:45.033254 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:45.033204    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:46.647418 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:46.647379    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:15:46.924951 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:46.924871    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:15:50.422784 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:15:50.422705    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:16:05.153106 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:05.153068    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:05.835734 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:05.835699    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9045-8c5871"]
Sep 17 11:16:05.839311 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:05.839293    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9045-8c5871"
Sep 17 11:16:05.842317 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:05.842294    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bnkhs\""
Sep 17 11:16:05.846016 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:05.845993    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9045-8c5871"]
Sep 17 11:16:05.896203 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:05.896158    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvkdg\" (UniqueName: \"kubernetes.io/projected/169be01f-ede0-450b-9e10-182c6daf7561-kube-api-access-rvkdg\") pod \"test-curl-9045-8c5871\" (UID: \"169be01f-ede0-450b-9e10-182c6daf7561\") " pod="opendatahub/test-curl-9045-8c5871"
Sep 17 11:16:05.997101 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:05.997061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvkdg\" (UniqueName: \"kubernetes.io/projected/169be01f-ede0-450b-9e10-182c6daf7561-kube-api-access-rvkdg\") pod \"test-curl-9045-8c5871\" (UID: \"169be01f-ede0-450b-9e10-182c6daf7561\") " pod="opendatahub/test-curl-9045-8c5871"
Sep 17 11:16:06.006068 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:06.006033    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvkdg\" (UniqueName: \"kubernetes.io/projected/169be01f-ede0-450b-9e10-182c6daf7561-kube-api-access-rvkdg\") pod \"test-curl-9045-8c5871\" (UID: \"169be01f-ede0-450b-9e10-182c6daf7561\") " pod="opendatahub/test-curl-9045-8c5871"
Sep 17 11:16:06.150851 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:06.150740    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9045-8c5871"
Sep 17 11:16:06.306026 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:06.306000    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:06.421410 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:06.421387    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9045-8c5871"]
Sep 17 11:16:06.423429 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:16:06.423394    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod169be01f_ede0_450b_9e10_182c6daf7561.slice/crio-36e5cdc83d627768e966e56fd95e07fa016f4708cad3abb653b5ccc09ce2d016 WatchSource:0}: Error finding container 36e5cdc83d627768e966e56fd95e07fa016f4708cad3abb653b5ccc09ce2d016: Status 404 returned error can't find the container with id 36e5cdc83d627768e966e56fd95e07fa016f4708cad3abb653b5ccc09ce2d016
Sep 17 11:16:06.473564 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:06.473543    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:06.513256 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:06.513220    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:16:06.520545 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:06.520514    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9045-8c5871" event={"ID":"169be01f-ede0-450b-9e10-182c6daf7561","Type":"ContainerStarted","Data":"9a73e96b38007b364defcc612a42c2570aef8b7d700966c37c0fe2acbee53d7b"}
Sep 17 11:16:06.520545 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:06.520552    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9045-8c5871" event={"ID":"169be01f-ede0-450b-9e10-182c6daf7561","Type":"ContainerStarted","Data":"36e5cdc83d627768e966e56fd95e07fa016f4708cad3abb653b5ccc09ce2d016"}
Sep 17 11:16:06.541797 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:06.541744    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9045-8c5871" podStartSLOduration=1.541725952 podStartE2EDuration="1.541725952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:16:06.539866747 +0000 UTC m=+1360.637912313" watchObservedRunningTime="2026-09-17 11:16:06.541725952 +0000 UTC m=+1360.639771517"
Sep 17 11:16:07.163002 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:07.162963    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9045-8c5871"]
Sep 17 11:16:07.167582 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:07.167555    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9045-8c5871"]
Sep 17 11:16:07.285431 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:07.285398    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9045-7035d3"]
Sep 17 11:16:07.285845 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:07.285830    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="169be01f-ede0-450b-9e10-182c6daf7561" containerName="test-curl-9045-8c5871"
Sep 17 11:16:07.285887 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:07.285847    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="169be01f-ede0-450b-9e10-182c6daf7561" containerName="test-curl-9045-8c5871"
Sep 17 11:16:07.285933 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:07.285927    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="169be01f-ede0-450b-9e10-182c6daf7561" containerName="test-curl-9045-8c5871"
Sep 17 11:16:07.289189 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:07.289173    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9045-7035d3"
Sep 17 11:16:07.304718 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:07.304684    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9045-7035d3"]
Sep 17 11:16:07.405523 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:07.405491    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:07.412698 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:07.412663    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xlm5\" (UniqueName: \"kubernetes.io/projected/d1cd967b-1512-4063-a479-de05e7bef058-kube-api-access-6xlm5\") pod \"test-curl-9045-7035d3\" (UID: \"d1cd967b-1512-4063-a479-de05e7bef058\") " pod="opendatahub/test-curl-9045-7035d3"
Sep 17 11:16:07.514201 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:07.514111    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xlm5\" (UniqueName: \"kubernetes.io/projected/d1cd967b-1512-4063-a479-de05e7bef058-kube-api-access-6xlm5\") pod \"test-curl-9045-7035d3\" (UID: \"d1cd967b-1512-4063-a479-de05e7bef058\") " pod="opendatahub/test-curl-9045-7035d3"
Sep 17 11:16:07.525766 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:07.525736    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xlm5\" (UniqueName: \"kubernetes.io/projected/d1cd967b-1512-4063-a479-de05e7bef058-kube-api-access-6xlm5\") pod \"test-curl-9045-7035d3\" (UID: \"d1cd967b-1512-4063-a479-de05e7bef058\") " pod="opendatahub/test-curl-9045-7035d3"
Sep 17 11:16:07.526015 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:16:07.525997    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9045-8c5871" container="test-curl-9045-8c5871"
Sep 17 11:16:07.598809 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:07.598773    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9045-7035d3"
Sep 17 11:16:07.869781 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:07.869757    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9045-7035d3"]
Sep 17 11:16:07.871787 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:16:07.871759    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1cd967b_1512_4063_a479_de05e7bef058.slice/crio-bab3a479b7ac983d96797ce6f6ae88933941076e5b3d43a0b8db4b2962d110e3 WatchSource:0}: Error finding container bab3a479b7ac983d96797ce6f6ae88933941076e5b3d43a0b8db4b2962d110e3: Status 404 returned error can't find the container with id bab3a479b7ac983d96797ce6f6ae88933941076e5b3d43a0b8db4b2962d110e3
Sep 17 11:16:08.028028 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:08.027985    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:08.531875 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:08.531836    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9045-7035d3" event={"ID":"d1cd967b-1512-4063-a479-de05e7bef058","Type":"ContainerStarted","Data":"f71093ee292e163172a1059d0eeb072e369632895606bc7e2fbcc00014adf897"}
Sep 17 11:16:08.532309 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:08.531886    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9045-7035d3" event={"ID":"d1cd967b-1512-4063-a479-de05e7bef058","Type":"ContainerStarted","Data":"bab3a479b7ac983d96797ce6f6ae88933941076e5b3d43a0b8db4b2962d110e3"}
Sep 17 11:16:08.532309 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:08.531908    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9045-8c5871" podUID="169be01f-ede0-450b-9e10-182c6daf7561" containerName="test-curl-9045-8c5871" containerID="cri-o://9a73e96b38007b364defcc612a42c2570aef8b7d700966c37c0fe2acbee53d7b" gracePeriod=2
Sep 17 11:16:08.548000 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:08.547936    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9045-7035d3" podStartSLOduration=1.547920429 podStartE2EDuration="1.547920429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:16:08.546809678 +0000 UTC m=+1362.644855246" watchObservedRunningTime="2026-09-17 11:16:08.547920429 +0000 UTC m=+1362.645965995"
Sep 17 11:16:09.233203 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:09.233167    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9045-7035d3"]
Sep 17 11:16:09.241303 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:09.241274    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9045-7035d3"]
Sep 17 11:16:09.359591 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:09.359553    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9045-640a2a"]
Sep 17 11:16:09.359977 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:09.359964    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1cd967b-1512-4063-a479-de05e7bef058" containerName="test-curl-9045-7035d3"
Sep 17 11:16:09.360064 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:09.359978    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1cd967b-1512-4063-a479-de05e7bef058" containerName="test-curl-9045-7035d3"
Sep 17 11:16:09.360118 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:09.360069    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1cd967b-1512-4063-a479-de05e7bef058" containerName="test-curl-9045-7035d3"
Sep 17 11:16:09.363203 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:09.363187    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9045-640a2a"
Sep 17 11:16:09.368848 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:09.368805    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9045-640a2a"]
Sep 17 11:16:09.433006 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:09.432967    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq8zz\" (UniqueName: \"kubernetes.io/projected/58701b8b-9b50-4b78-89c8-243d80f13d20-kube-api-access-qq8zz\") pod \"test-curl-9045-640a2a\" (UID: \"58701b8b-9b50-4b78-89c8-243d80f13d20\") " pod="opendatahub/test-curl-9045-640a2a"
Sep 17 11:16:09.533751 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:09.533710    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq8zz\" (UniqueName: \"kubernetes.io/projected/58701b8b-9b50-4b78-89c8-243d80f13d20-kube-api-access-qq8zz\") pod \"test-curl-9045-640a2a\" (UID: \"58701b8b-9b50-4b78-89c8-243d80f13d20\") " pod="opendatahub/test-curl-9045-640a2a"
Sep 17 11:16:09.536944 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:16:09.536920    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9045-7035d3" container="test-curl-9045-7035d3"
Sep 17 11:16:09.544148 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:09.544124    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq8zz\" (UniqueName: \"kubernetes.io/projected/58701b8b-9b50-4b78-89c8-243d80f13d20-kube-api-access-qq8zz\") pod \"test-curl-9045-640a2a\" (UID: \"58701b8b-9b50-4b78-89c8-243d80f13d20\") " pod="opendatahub/test-curl-9045-640a2a"
Sep 17 11:16:09.673406 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:09.673327    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9045-640a2a"
Sep 17 11:16:09.951701 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:09.951672    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9045-640a2a"]
Sep 17 11:16:09.953936 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:16:09.953905    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58701b8b_9b50_4b78_89c8_243d80f13d20.slice/crio-833b97276634d5eb41cef9740cab52619dc448a34abba0480a4a46c713918c86 WatchSource:0}: Error finding container 833b97276634d5eb41cef9740cab52619dc448a34abba0480a4a46c713918c86: Status 404 returned error can't find the container with id 833b97276634d5eb41cef9740cab52619dc448a34abba0480a4a46c713918c86
Sep 17 11:16:10.543459 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:10.543425    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9045-640a2a" event={"ID":"58701b8b-9b50-4b78-89c8-243d80f13d20","Type":"ContainerStarted","Data":"8aec0acc0e1d73e4ecc90dce45371148859aba4d5f4c0aa1fe453a0ceeca9bd3"}
Sep 17 11:16:10.543834 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:10.543472    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9045-640a2a" event={"ID":"58701b8b-9b50-4b78-89c8-243d80f13d20","Type":"ContainerStarted","Data":"833b97276634d5eb41cef9740cab52619dc448a34abba0480a4a46c713918c86"}
Sep 17 11:16:10.543834 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:10.543593    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9045-7035d3" podUID="d1cd967b-1512-4063-a479-de05e7bef058" containerName="test-curl-9045-7035d3" containerID="cri-o://f71093ee292e163172a1059d0eeb072e369632895606bc7e2fbcc00014adf897" gracePeriod=2
Sep 17 11:16:10.560485 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:10.560429    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9045-640a2a" podStartSLOduration=1.560413791 podStartE2EDuration="1.560413791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:16:10.558217513 +0000 UTC m=+1364.656263079" watchObservedRunningTime="2026-09-17 11:16:10.560413791 +0000 UTC m=+1364.658459357"
Sep 17 11:16:10.729150 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:10.729119    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9045-8c5871"
Sep 17 11:16:10.844903 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:10.844784    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvkdg\" (UniqueName: \"kubernetes.io/projected/169be01f-ede0-450b-9e10-182c6daf7561-kube-api-access-rvkdg\") pod \"169be01f-ede0-450b-9e10-182c6daf7561\" (UID: \"169be01f-ede0-450b-9e10-182c6daf7561\") "
Sep 17 11:16:10.846965 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:10.846938    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169be01f-ede0-450b-9e10-182c6daf7561-kube-api-access-rvkdg" (OuterVolumeSpecName: "kube-api-access-rvkdg") pod "169be01f-ede0-450b-9e10-182c6daf7561" (UID: "169be01f-ede0-450b-9e10-182c6daf7561"). InnerVolumeSpecName "kube-api-access-rvkdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:16:10.946367 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:10.946337    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvkdg\" (UniqueName: \"kubernetes.io/projected/169be01f-ede0-450b-9e10-182c6daf7561-kube-api-access-rvkdg\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:16:11.133411 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:11.133322    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9045-640a2a"]
Sep 17 11:16:11.142302 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:11.142274    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9045-640a2a"]
Sep 17 11:16:11.252015 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:11.251966    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9045-4bd0cf"]
Sep 17 11:16:11.252382 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:11.252369    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58701b8b-9b50-4b78-89c8-243d80f13d20" containerName="test-curl-9045-640a2a"
Sep 17 11:16:11.252428 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:11.252383    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="58701b8b-9b50-4b78-89c8-243d80f13d20" containerName="test-curl-9045-640a2a"
Sep 17 11:16:11.252488 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:11.252479    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="58701b8b-9b50-4b78-89c8-243d80f13d20" containerName="test-curl-9045-640a2a"
Sep 17 11:16:11.255589 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:11.255572    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9045-4bd0cf"
Sep 17 11:16:11.261190 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:11.261158    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9045-4bd0cf"]
Sep 17 11:16:11.350517 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:11.350483    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppxbh\" (UniqueName: \"kubernetes.io/projected/f482ff62-c31c-4630-bcca-8724729bb4fa-kube-api-access-ppxbh\") pod \"test-curl-9045-4bd0cf\" (UID: \"f482ff62-c31c-4630-bcca-8724729bb4fa\") " pod="opendatahub/test-curl-9045-4bd0cf"
Sep 17 11:16:11.452052 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:11.451961    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppxbh\" (UniqueName: \"kubernetes.io/projected/f482ff62-c31c-4630-bcca-8724729bb4fa-kube-api-access-ppxbh\") pod \"test-curl-9045-4bd0cf\" (UID: \"f482ff62-c31c-4630-bcca-8724729bb4fa\") " pod="opendatahub/test-curl-9045-4bd0cf"
Sep 17 11:16:11.460716 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:11.460689    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppxbh\" (UniqueName: \"kubernetes.io/projected/f482ff62-c31c-4630-bcca-8724729bb4fa-kube-api-access-ppxbh\") pod \"test-curl-9045-4bd0cf\" (UID: \"f482ff62-c31c-4630-bcca-8724729bb4fa\") " pod="opendatahub/test-curl-9045-4bd0cf"
Sep 17 11:16:11.550649 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:11.550616    2562 generic.go:356] "Generic (PLEG): container finished" podID="169be01f-ede0-450b-9e10-182c6daf7561" containerID="9a73e96b38007b364defcc612a42c2570aef8b7d700966c37c0fe2acbee53d7b" exitCode=137
Sep 17 11:16:11.551100 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:11.550656    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9045-8c5871"
Sep 17 11:16:11.551100 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:11.550703    2562 scope.go:117] "RemoveContainer" containerID="9a73e96b38007b364defcc612a42c2570aef8b7d700966c37c0fe2acbee53d7b"
Sep 17 11:16:11.551100 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:16:11.551019    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9045-640a2a" container="test-curl-9045-640a2a"
Sep 17 11:16:11.559990 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:11.559972    2562 scope.go:117] "RemoveContainer" containerID="9a73e96b38007b364defcc612a42c2570aef8b7d700966c37c0fe2acbee53d7b"
Sep 17 11:16:11.560290 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:16:11.560271    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a73e96b38007b364defcc612a42c2570aef8b7d700966c37c0fe2acbee53d7b\": container with ID starting with 9a73e96b38007b364defcc612a42c2570aef8b7d700966c37c0fe2acbee53d7b not found: ID does not exist" containerID="9a73e96b38007b364defcc612a42c2570aef8b7d700966c37c0fe2acbee53d7b"
Sep 17 11:16:11.560356 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:11.560302    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a73e96b38007b364defcc612a42c2570aef8b7d700966c37c0fe2acbee53d7b"} err="failed to get container status \"9a73e96b38007b364defcc612a42c2570aef8b7d700966c37c0fe2acbee53d7b\": rpc error: code = NotFound desc = could not find container \"9a73e96b38007b364defcc612a42c2570aef8b7d700966c37c0fe2acbee53d7b\": container with ID starting with 9a73e96b38007b364defcc612a42c2570aef8b7d700966c37c0fe2acbee53d7b not found: ID does not exist"
Sep 17 11:16:11.561996 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:16:11.561971    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9045-8c5871\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="169be01f-ede0-450b-9e10-182c6daf7561" pod="opendatahub/test-curl-9045-8c5871"
Sep 17 11:16:11.567186 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:11.567166    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9045-4bd0cf"
Sep 17 11:16:11.842078 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:11.842043    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9045-4bd0cf"]
Sep 17 11:16:11.844915 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:16:11.844889    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf482ff62_c31c_4630_bcca_8724729bb4fa.slice/crio-4b6e0ab4d9d996469748a9705db0ffebda7c094d3d74274fd36dc0e482d68de7 WatchSource:0}: Error finding container 4b6e0ab4d9d996469748a9705db0ffebda7c094d3d74274fd36dc0e482d68de7: Status 404 returned error can't find the container with id 4b6e0ab4d9d996469748a9705db0ffebda7c094d3d74274fd36dc0e482d68de7
Sep 17 11:16:12.455883 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:12.455851    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="169be01f-ede0-450b-9e10-182c6daf7561" path="/var/lib/kubelet/pods/169be01f-ede0-450b-9e10-182c6daf7561/volumes"
Sep 17 11:16:12.557605 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:12.557575    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9045-4bd0cf" event={"ID":"f482ff62-c31c-4630-bcca-8724729bb4fa","Type":"ContainerStarted","Data":"98fad4a870f8ac6b2fbf4bae9febbc51950ee8deaed73a3602a4f7ac43fd5459"}
Sep 17 11:16:12.557984 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:12.557618    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9045-4bd0cf" event={"ID":"f482ff62-c31c-4630-bcca-8724729bb4fa","Type":"ContainerStarted","Data":"4b6e0ab4d9d996469748a9705db0ffebda7c094d3d74274fd36dc0e482d68de7"}
Sep 17 11:16:12.558779 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:12.558669    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9045-640a2a" podUID="58701b8b-9b50-4b78-89c8-243d80f13d20" containerName="test-curl-9045-640a2a" containerID="cri-o://8aec0acc0e1d73e4ecc90dce45371148859aba4d5f4c0aa1fe453a0ceeca9bd3" gracePeriod=2
Sep 17 11:16:12.572994 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:12.572940    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9045-4bd0cf" podStartSLOduration=1.572919414 podStartE2EDuration="1.572919414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:16:12.572528382 +0000 UTC m=+1366.670573949" watchObservedRunningTime="2026-09-17 11:16:12.572919414 +0000 UTC m=+1366.670964982"
Sep 17 11:16:12.718402 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:12.718331    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9045-7035d3"
Sep 17 11:16:12.865729 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:12.865692    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xlm5\" (UniqueName: \"kubernetes.io/projected/d1cd967b-1512-4063-a479-de05e7bef058-kube-api-access-6xlm5\") pod \"d1cd967b-1512-4063-a479-de05e7bef058\" (UID: \"d1cd967b-1512-4063-a479-de05e7bef058\") "
Sep 17 11:16:12.867806 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:12.867774    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1cd967b-1512-4063-a479-de05e7bef058-kube-api-access-6xlm5" (OuterVolumeSpecName: "kube-api-access-6xlm5") pod "d1cd967b-1512-4063-a479-de05e7bef058" (UID: "d1cd967b-1512-4063-a479-de05e7bef058"). InnerVolumeSpecName "kube-api-access-6xlm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:16:12.966659 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:12.966635    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xlm5\" (UniqueName: \"kubernetes.io/projected/d1cd967b-1512-4063-a479-de05e7bef058-kube-api-access-6xlm5\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:16:13.175244 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:13.175207    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9045-4bd0cf"]
Sep 17 11:16:13.178280 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:13.178255    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9045-4bd0cf"]
Sep 17 11:16:13.565348 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:13.565313    2562 generic.go:356] "Generic (PLEG): container finished" podID="d1cd967b-1512-4063-a479-de05e7bef058" containerID="f71093ee292e163172a1059d0eeb072e369632895606bc7e2fbcc00014adf897" exitCode=137
Sep 17 11:16:13.565794 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:13.565352    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9045-7035d3"
Sep 17 11:16:13.565794 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:13.565407    2562 scope.go:117] "RemoveContainer" containerID="f71093ee292e163172a1059d0eeb072e369632895606bc7e2fbcc00014adf897"
Sep 17 11:16:13.565794 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:16:13.565605    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9045-4bd0cf" container="test-curl-9045-4bd0cf"
Sep 17 11:16:13.577003 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:16:13.576965    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9045-7035d3\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="d1cd967b-1512-4063-a479-de05e7bef058" pod="opendatahub/test-curl-9045-7035d3"
Sep 17 11:16:13.578870 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:13.578848    2562 scope.go:117] "RemoveContainer" containerID="f71093ee292e163172a1059d0eeb072e369632895606bc7e2fbcc00014adf897"
Sep 17 11:16:13.579157 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:16:13.579136    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71093ee292e163172a1059d0eeb072e369632895606bc7e2fbcc00014adf897\": container with ID starting with f71093ee292e163172a1059d0eeb072e369632895606bc7e2fbcc00014adf897 not found: ID does not exist" containerID="f71093ee292e163172a1059d0eeb072e369632895606bc7e2fbcc00014adf897"
Sep 17 11:16:13.579214 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:13.579164    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71093ee292e163172a1059d0eeb072e369632895606bc7e2fbcc00014adf897"} err="failed to get container status \"f71093ee292e163172a1059d0eeb072e369632895606bc7e2fbcc00014adf897\": rpc error: code = NotFound desc = could not find container \"f71093ee292e163172a1059d0eeb072e369632895606bc7e2fbcc00014adf897\": container with ID starting with f71093ee292e163172a1059d0eeb072e369632895606bc7e2fbcc00014adf897 not found: ID does not exist"
Sep 17 11:16:13.781121 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:13.781086    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:14.116428 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:14.116385    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:14.456272 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:14.456185    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1cd967b-1512-4063-a479-de05e7bef058" path="/var/lib/kubelet/pods/d1cd967b-1512-4063-a479-de05e7bef058/volumes"
Sep 17 11:16:14.571223 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:14.571195    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9045-4bd0cf" podUID="f482ff62-c31c-4630-bcca-8724729bb4fa" containerName="test-curl-9045-4bd0cf" containerID="cri-o://98fad4a870f8ac6b2fbf4bae9febbc51950ee8deaed73a3602a4f7ac43fd5459" gracePeriod=2
Sep 17 11:16:14.697297 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:14.697268    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:14.749233 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:14.749206    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9045-640a2a"
Sep 17 11:16:14.884664 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:14.884636    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qq8zz\" (UniqueName: \"kubernetes.io/projected/58701b8b-9b50-4b78-89c8-243d80f13d20-kube-api-access-qq8zz\") pod \"58701b8b-9b50-4b78-89c8-243d80f13d20\" (UID: \"58701b8b-9b50-4b78-89c8-243d80f13d20\") "
Sep 17 11:16:14.886684 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:14.886660    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58701b8b-9b50-4b78-89c8-243d80f13d20-kube-api-access-qq8zz" (OuterVolumeSpecName: "kube-api-access-qq8zz") pod "58701b8b-9b50-4b78-89c8-243d80f13d20" (UID: "58701b8b-9b50-4b78-89c8-243d80f13d20"). InnerVolumeSpecName "kube-api-access-qq8zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:16:14.986108 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:14.986018    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq8zz\" (UniqueName: \"kubernetes.io/projected/58701b8b-9b50-4b78-89c8-243d80f13d20-kube-api-access-qq8zz\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:16:15.577774 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:15.577733    2562 generic.go:356] "Generic (PLEG): container finished" podID="58701b8b-9b50-4b78-89c8-243d80f13d20" containerID="8aec0acc0e1d73e4ecc90dce45371148859aba4d5f4c0aa1fe453a0ceeca9bd3" exitCode=137
Sep 17 11:16:15.578231 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:15.577785    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9045-640a2a"
Sep 17 11:16:15.578231 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:15.577852    2562 scope.go:117] "RemoveContainer" containerID="8aec0acc0e1d73e4ecc90dce45371148859aba4d5f4c0aa1fe453a0ceeca9bd3"
Sep 17 11:16:15.586583 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:15.586567    2562 scope.go:117] "RemoveContainer" containerID="8aec0acc0e1d73e4ecc90dce45371148859aba4d5f4c0aa1fe453a0ceeca9bd3"
Sep 17 11:16:15.586891 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:16:15.586871    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aec0acc0e1d73e4ecc90dce45371148859aba4d5f4c0aa1fe453a0ceeca9bd3\": container with ID starting with 8aec0acc0e1d73e4ecc90dce45371148859aba4d5f4c0aa1fe453a0ceeca9bd3 not found: ID does not exist" containerID="8aec0acc0e1d73e4ecc90dce45371148859aba4d5f4c0aa1fe453a0ceeca9bd3"
Sep 17 11:16:15.586938 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:15.586901    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aec0acc0e1d73e4ecc90dce45371148859aba4d5f4c0aa1fe453a0ceeca9bd3"} err="failed to get container status \"8aec0acc0e1d73e4ecc90dce45371148859aba4d5f4c0aa1fe453a0ceeca9bd3\": rpc error: code = NotFound desc = could not find container \"8aec0acc0e1d73e4ecc90dce45371148859aba4d5f4c0aa1fe453a0ceeca9bd3\": container with ID starting with 8aec0acc0e1d73e4ecc90dce45371148859aba4d5f4c0aa1fe453a0ceeca9bd3 not found: ID does not exist"
Sep 17 11:16:15.588310 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:16:15.588290    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9045-640a2a\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="58701b8b-9b50-4b78-89c8-243d80f13d20" pod="opendatahub/test-curl-9045-640a2a"
Sep 17 11:16:16.454207 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:16:16.454168    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9045-640a2a\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="58701b8b-9b50-4b78-89c8-243d80f13d20" pod="opendatahub/test-curl-9045-640a2a"
Sep 17 11:16:16.455694 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:16.455673    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58701b8b-9b50-4b78-89c8-243d80f13d20" path="/var/lib/kubelet/pods/58701b8b-9b50-4b78-89c8-243d80f13d20/volumes"
Sep 17 11:16:16.745974 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:16.745944    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9045-4bd0cf"
Sep 17 11:16:16.903786 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:16.903749    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppxbh\" (UniqueName: \"kubernetes.io/projected/f482ff62-c31c-4630-bcca-8724729bb4fa-kube-api-access-ppxbh\") pod \"f482ff62-c31c-4630-bcca-8724729bb4fa\" (UID: \"f482ff62-c31c-4630-bcca-8724729bb4fa\") "
Sep 17 11:16:16.905789 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:16.905764    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f482ff62-c31c-4630-bcca-8724729bb4fa-kube-api-access-ppxbh" (OuterVolumeSpecName: "kube-api-access-ppxbh") pod "f482ff62-c31c-4630-bcca-8724729bb4fa" (UID: "f482ff62-c31c-4630-bcca-8724729bb4fa"). InnerVolumeSpecName "kube-api-access-ppxbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:16:17.005058 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:17.004970    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppxbh\" (UniqueName: \"kubernetes.io/projected/f482ff62-c31c-4630-bcca-8724729bb4fa-kube-api-access-ppxbh\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:16:17.590380 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:17.590349    2562 generic.go:356] "Generic (PLEG): container finished" podID="f482ff62-c31c-4630-bcca-8724729bb4fa" containerID="98fad4a870f8ac6b2fbf4bae9febbc51950ee8deaed73a3602a4f7ac43fd5459" exitCode=137
Sep 17 11:16:17.590542 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:17.590390    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9045-4bd0cf"
Sep 17 11:16:17.590542 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:17.590455    2562 scope.go:117] "RemoveContainer" containerID="98fad4a870f8ac6b2fbf4bae9febbc51950ee8deaed73a3602a4f7ac43fd5459"
Sep 17 11:16:17.599737 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:17.599720    2562 scope.go:117] "RemoveContainer" containerID="98fad4a870f8ac6b2fbf4bae9febbc51950ee8deaed73a3602a4f7ac43fd5459"
Sep 17 11:16:17.600023 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:16:17.600006    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98fad4a870f8ac6b2fbf4bae9febbc51950ee8deaed73a3602a4f7ac43fd5459\": container with ID starting with 98fad4a870f8ac6b2fbf4bae9febbc51950ee8deaed73a3602a4f7ac43fd5459 not found: ID does not exist" containerID="98fad4a870f8ac6b2fbf4bae9febbc51950ee8deaed73a3602a4f7ac43fd5459"
Sep 17 11:16:17.600074 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:17.600032    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98fad4a870f8ac6b2fbf4bae9febbc51950ee8deaed73a3602a4f7ac43fd5459"} err="failed to get container status \"98fad4a870f8ac6b2fbf4bae9febbc51950ee8deaed73a3602a4f7ac43fd5459\": rpc error: code = NotFound desc = could not find container \"98fad4a870f8ac6b2fbf4bae9febbc51950ee8deaed73a3602a4f7ac43fd5459\": container with ID starting with 98fad4a870f8ac6b2fbf4bae9febbc51950ee8deaed73a3602a4f7ac43fd5459 not found: ID does not exist"
Sep 17 11:16:17.603425 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:16:17.603406    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9045-4bd0cf\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="f482ff62-c31c-4630-bcca-8724729bb4fa" pod="opendatahub/test-curl-9045-4bd0cf"
Sep 17 11:16:17.824655 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:17.824608    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:17.926209 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:17.926079    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:18.434805 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:18.434768    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:18.456213 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:18.456180    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f482ff62-c31c-4630-bcca-8724729bb4fa" path="/var/lib/kubelet/pods/f482ff62-c31c-4630-bcca-8724729bb4fa/volumes"
Sep 17 11:16:19.166513 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:19.166475    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:20.046774 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:20.046742    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:21.022186 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.022153    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"]
Sep 17 11:16:21.022673 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.022543    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f482ff62-c31c-4630-bcca-8724729bb4fa" containerName="test-curl-9045-4bd0cf"
Sep 17 11:16:21.022673 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.022555    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f482ff62-c31c-4630-bcca-8724729bb4fa" containerName="test-curl-9045-4bd0cf"
Sep 17 11:16:21.022673 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.022623    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f482ff62-c31c-4630-bcca-8724729bb4fa" containerName="test-curl-9045-4bd0cf"
Sep 17 11:16:21.034464 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.034437    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:16:21.037433 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.037411    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d727624f-serving-cert\""
Sep 17 11:16:21.040424 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.040393    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"]
Sep 17 11:16:21.059141 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.059104    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:21.140582 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.140535    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/982b56f5-c2b6-4f78-bc8a-ce51d455c631-openshift-service-ca\") pod \"maas-api-e2e-mt-d727624f-5f9988845f-hgvv8\" (UID: \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:16:21.140582 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.140587    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982b56f5-c2b6-4f78-bc8a-ce51d455c631-metrics-certs\") pod \"maas-api-e2e-mt-d727624f-5f9988845f-hgvv8\" (UID: \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:16:21.140812 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.140671    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982b56f5-c2b6-4f78-bc8a-ce51d455c631-maas-api-tls\") pod \"maas-api-e2e-mt-d727624f-5f9988845f-hgvv8\" (UID: \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:16:21.140812 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.140702    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54vvt\" (UniqueName: \"kubernetes.io/projected/982b56f5-c2b6-4f78-bc8a-ce51d455c631-kube-api-access-54vvt\") pod \"maas-api-e2e-mt-d727624f-5f9988845f-hgvv8\" (UID: \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:16:21.143317 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.143294    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:21.242133 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.242099    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/982b56f5-c2b6-4f78-bc8a-ce51d455c631-openshift-service-ca\") pod \"maas-api-e2e-mt-d727624f-5f9988845f-hgvv8\" (UID: \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:16:21.242296 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.242138    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982b56f5-c2b6-4f78-bc8a-ce51d455c631-metrics-certs\") pod \"maas-api-e2e-mt-d727624f-5f9988845f-hgvv8\" (UID: \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:16:21.242296 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.242188    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982b56f5-c2b6-4f78-bc8a-ce51d455c631-maas-api-tls\") pod \"maas-api-e2e-mt-d727624f-5f9988845f-hgvv8\" (UID: \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:16:21.242296 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.242208    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54vvt\" (UniqueName: \"kubernetes.io/projected/982b56f5-c2b6-4f78-bc8a-ce51d455c631-kube-api-access-54vvt\") pod \"maas-api-e2e-mt-d727624f-5f9988845f-hgvv8\" (UID: \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:16:21.242462 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:16:21.242310    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-serving-cert: secret "maas-api-e2e-mt-d727624f-serving-cert" not found
Sep 17 11:16:21.242462 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:16:21.242381    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982b56f5-c2b6-4f78-bc8a-ce51d455c631-maas-api-tls podName:982b56f5-c2b6-4f78-bc8a-ce51d455c631 nodeName:}" failed. No retries permitted until 2026-09-17 11:16:21.74236285 +0000 UTC m=+1375.840408399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/982b56f5-c2b6-4f78-bc8a-ce51d455c631-maas-api-tls") pod "maas-api-e2e-mt-d727624f-5f9988845f-hgvv8" (UID: "982b56f5-c2b6-4f78-bc8a-ce51d455c631") : secret "maas-api-e2e-mt-d727624f-serving-cert" not found
Sep 17 11:16:21.242885 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.242851    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/982b56f5-c2b6-4f78-bc8a-ce51d455c631-openshift-service-ca\") pod \"maas-api-e2e-mt-d727624f-5f9988845f-hgvv8\" (UID: \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:16:21.244761 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.244738    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982b56f5-c2b6-4f78-bc8a-ce51d455c631-metrics-certs\") pod \"maas-api-e2e-mt-d727624f-5f9988845f-hgvv8\" (UID: \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:16:21.251920 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.251891    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54vvt\" (UniqueName: \"kubernetes.io/projected/982b56f5-c2b6-4f78-bc8a-ce51d455c631-kube-api-access-54vvt\") pod \"maas-api-e2e-mt-d727624f-5f9988845f-hgvv8\" (UID: \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:16:21.747388 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.747349    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982b56f5-c2b6-4f78-bc8a-ce51d455c631-maas-api-tls\") pod \"maas-api-e2e-mt-d727624f-5f9988845f-hgvv8\" (UID: \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:16:21.747551 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:16:21.747498    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-serving-cert: secret "maas-api-e2e-mt-d727624f-serving-cert" not found
Sep 17 11:16:21.747590 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:16:21.747579    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982b56f5-c2b6-4f78-bc8a-ce51d455c631-maas-api-tls podName:982b56f5-c2b6-4f78-bc8a-ce51d455c631 nodeName:}" failed. No retries permitted until 2026-09-17 11:16:22.747562915 +0000 UTC m=+1376.845608458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/982b56f5-c2b6-4f78-bc8a-ce51d455c631-maas-api-tls") pod "maas-api-e2e-mt-d727624f-5f9988845f-hgvv8" (UID: "982b56f5-c2b6-4f78-bc8a-ce51d455c631") : secret "maas-api-e2e-mt-d727624f-serving-cert" not found
Sep 17 11:16:21.761664 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.761632    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:21.884295 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:21.884262    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:22.757409 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:22.757371    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982b56f5-c2b6-4f78-bc8a-ce51d455c631-maas-api-tls\") pod \"maas-api-e2e-mt-d727624f-5f9988845f-hgvv8\" (UID: \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:16:22.759918 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:22.759892    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982b56f5-c2b6-4f78-bc8a-ce51d455c631-maas-api-tls\") pod \"maas-api-e2e-mt-d727624f-5f9988845f-hgvv8\" (UID: \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:16:22.847257 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:22.847221    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:16:22.851442 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:22.851418    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:23.054463 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:23.054424    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:16:23.061908 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:23.061873    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:23.125667 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:23.125632    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"]
Sep 17 11:16:23.128687 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:16:23.128660    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982b56f5_c2b6_4f78_bc8a_ce51d455c631.slice/crio-e7d75634b0d6b701edcc9439d694ef878bb19128259b78442d7b0b2707acf4c8 WatchSource:0}: Error finding container e7d75634b0d6b701edcc9439d694ef878bb19128259b78442d7b0b2707acf4c8: Status 404 returned error can't find the container with id e7d75634b0d6b701edcc9439d694ef878bb19128259b78442d7b0b2707acf4c8
Sep 17 11:16:23.130085 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:23.130070    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 11:16:23.623059 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:23.623029    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8" event={"ID":"982b56f5-c2b6-4f78-bc8a-ce51d455c631","Type":"ContainerStarted","Data":"e7d75634b0d6b701edcc9439d694ef878bb19128259b78442d7b0b2707acf4c8"}
Sep 17 11:16:23.705009 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:23.704975    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:24.629762 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:24.629726    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8" event={"ID":"982b56f5-c2b6-4f78-bc8a-ce51d455c631","Type":"ContainerStarted","Data":"a5245fe408e1f7d705fbbe7f6c23041c80cd25c3187a1f55d5569f0a447851e8"}
Sep 17 11:16:24.630278 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:24.629831    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:16:24.649117 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:24.649062    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8" podStartSLOduration=3.165145185 podStartE2EDuration="3.649045472s" totalImagesPullingTime="483.900287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:16:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:16:24.64703052 +0000 UTC m=+1378.745076085" watchObservedRunningTime="2026-09-17 11:16:24.649045472 +0000 UTC m=+1378.747091039"
Sep 17 11:16:30.280004 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:30.279970    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:30.391375 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:30.391329    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:30.640162 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:30.640082    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:16:37.811170 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:37.811133    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:16:43.686832 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:43.686783    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:44.882535 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:44.882491    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:45.973904 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:45.973868    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:47.344239 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:47.344209    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:47.886249 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:47.886218    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:48.738437 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:48.738404    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:56.717069 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:56.717031    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:16:56.845662 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:56.845627    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"]
Sep 17 11:16:56.849032 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:56.849015    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:56.852370 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:56.852349    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d727624f\"/\"openshift-service-ca.crt\""
Sep 17 11:16:56.853251 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:56.853228    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d727624f\"/\"default-dockercfg-2hvpn\""
Sep 17 11:16:56.853251 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:56.853243    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d727624f\"/\"e2e-lifecycle-model-d727624f-kserve-self-signed-certs\""
Sep 17 11:16:56.853421 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:56.853254    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d727624f\"/\"kube-root-ca.crt\""
Sep 17 11:16:56.868509 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:56.868487    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"]
Sep 17 11:16:56.882446 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:56.882418    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:16:56.973489 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:56.973404    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b51be080-dc53-4c7f-aa57-edc475c5679e-tls-certs\") pod \"e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") " pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:56.973489 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:56.973446    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-dshm\") pod \"e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") " pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:56.973489 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:56.973478    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-home\") pod \"e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") " pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:56.973710 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:56.973522    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-tmp-dir\") pod \"e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") " pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:56.973710 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:56.973539    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-model-cache\") pod \"e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") " pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:56.973710 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:56.973571    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npvmp\" (UniqueName: \"kubernetes.io/projected/b51be080-dc53-4c7f-aa57-edc475c5679e-kube-api-access-npvmp\") pod \"e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") " pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:57.074731 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:57.074689    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npvmp\" (UniqueName: \"kubernetes.io/projected/b51be080-dc53-4c7f-aa57-edc475c5679e-kube-api-access-npvmp\") pod \"e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") " pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:57.074925 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:57.074757    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b51be080-dc53-4c7f-aa57-edc475c5679e-tls-certs\") pod \"e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") " pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:57.074925 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:57.074787    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-dshm\") pod \"e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") " pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:57.074925 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:57.074844    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-home\") pod \"e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") " pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:57.074925 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:57.074891    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-tmp-dir\") pod \"e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") " pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:57.074925 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:57.074908    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-model-cache\") pod \"e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") " pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:57.075342 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:57.075308    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-model-cache\") pod \"e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") " pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:57.075342 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:57.075328    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-home\") pod \"e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") " pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:57.075541 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:57.075361    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-tmp-dir\") pod \"e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") " pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:57.077085 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:57.077050    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-dshm\") pod \"e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") " pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:57.077347 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:57.077330    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b51be080-dc53-4c7f-aa57-edc475c5679e-tls-certs\") pod \"e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") " pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:57.087472 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:57.087446    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npvmp\" (UniqueName: \"kubernetes.io/projected/b51be080-dc53-4c7f-aa57-edc475c5679e-kube-api-access-npvmp\") pod \"e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") " pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:57.159167 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:57.159131    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:16:57.464268 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:57.464243    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"]
Sep 17 11:16:57.465548 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:16:57.465524    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb51be080_dc53_4c7f_aa57_edc475c5679e.slice/crio-c3e4e7b70b144883f68bbc0fdd76034d8c73a2eaf9d97ea99287cf35dfd850a5 WatchSource:0}: Error finding container c3e4e7b70b144883f68bbc0fdd76034d8c73a2eaf9d97ea99287cf35dfd850a5: Status 404 returned error can't find the container with id c3e4e7b70b144883f68bbc0fdd76034d8c73a2eaf9d97ea99287cf35dfd850a5
Sep 17 11:16:57.802778 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:57.802748    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq" event={"ID":"b51be080-dc53-4c7f-aa57-edc475c5679e","Type":"ContainerStarted","Data":"bc7056c8656df707dbe45b7651159534e171349d47d48e7add4d47b5e3dd4112"}
Sep 17 11:16:57.802778 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:57.802787    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq" event={"ID":"b51be080-dc53-4c7f-aa57-edc475c5679e","Type":"ContainerStarted","Data":"c3e4e7b70b144883f68bbc0fdd76034d8c73a2eaf9d97ea99287cf35dfd850a5"}
Sep 17 11:16:57.846028 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:57.845975    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq" podStartSLOduration=1.845960343 podStartE2EDuration="1.845960343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:16:57.844491891 +0000 UTC m=+1411.942537456" watchObservedRunningTime="2026-09-17 11:16:57.845960343 +0000 UTC m=+1411.944005901"
Sep 17 11:16:58.922371 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:58.922334    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9048-38c11b"]
Sep 17 11:16:58.926165 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:58.926141    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9048-38c11b"
Sep 17 11:16:58.934767 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:58.934744    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bnkhs\""
Sep 17 11:16:58.948808 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:58.948780    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9048-38c11b"]
Sep 17 11:16:58.992180 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:58.992146    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42dvz\" (UniqueName: \"kubernetes.io/projected/cc4a483c-c6e9-4588-8971-a53675b15da1-kube-api-access-42dvz\") pod \"test-curl-9048-38c11b\" (UID: \"cc4a483c-c6e9-4588-8971-a53675b15da1\") " pod="opendatahub/test-curl-9048-38c11b"
Sep 17 11:16:59.093117 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:59.093068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42dvz\" (UniqueName: \"kubernetes.io/projected/cc4a483c-c6e9-4588-8971-a53675b15da1-kube-api-access-42dvz\") pod \"test-curl-9048-38c11b\" (UID: \"cc4a483c-c6e9-4588-8971-a53675b15da1\") " pod="opendatahub/test-curl-9048-38c11b"
Sep 17 11:16:59.103736 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:59.103704    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42dvz\" (UniqueName: \"kubernetes.io/projected/cc4a483c-c6e9-4588-8971-a53675b15da1-kube-api-access-42dvz\") pod \"test-curl-9048-38c11b\" (UID: \"cc4a483c-c6e9-4588-8971-a53675b15da1\") " pod="opendatahub/test-curl-9048-38c11b"
Sep 17 11:16:59.235541 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:59.235452    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9048-38c11b"
Sep 17 11:16:59.519134 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:59.519110    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9048-38c11b"]
Sep 17 11:16:59.520757 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:16:59.520727    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc4a483c_c6e9_4588_8971_a53675b15da1.slice/crio-098193be576c6ea93d64dce9990cd5a73657686a576ec410f289ff3114cd77b7 WatchSource:0}: Error finding container 098193be576c6ea93d64dce9990cd5a73657686a576ec410f289ff3114cd77b7: Status 404 returned error can't find the container with id 098193be576c6ea93d64dce9990cd5a73657686a576ec410f289ff3114cd77b7
Sep 17 11:16:59.814968 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:59.814931    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9048-38c11b" event={"ID":"cc4a483c-c6e9-4588-8971-a53675b15da1","Type":"ContainerStarted","Data":"2fca1d492a66760c3de23f9b4b5cd1af7f2798fe51248c7683ce6fdf2e6ef923"}
Sep 17 11:16:59.815142 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:59.814977    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9048-38c11b" event={"ID":"cc4a483c-c6e9-4588-8971-a53675b15da1","Type":"ContainerStarted","Data":"098193be576c6ea93d64dce9990cd5a73657686a576ec410f289ff3114cd77b7"}
Sep 17 11:16:59.835017 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:16:59.834967    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9048-38c11b" podStartSLOduration=1.8349515570000001 podStartE2EDuration="1.834951557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:16:59.834067112 +0000 UTC m=+1413.932112680" watchObservedRunningTime="2026-09-17 11:16:59.834951557 +0000 UTC m=+1413.932997123"
Sep 17 11:17:00.634969 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:00.634933    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9048-38c11b"]
Sep 17 11:17:00.640927 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:00.640893    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9048-38c11b"]
Sep 17 11:17:00.820267 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:17:00.820237    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9048-38c11b" container="test-curl-9048-38c11b"
Sep 17 11:17:01.825554 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:01.825514    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9048-38c11b" podUID="cc4a483c-c6e9-4588-8971-a53675b15da1" containerName="test-curl-9048-38c11b" containerID="cri-o://2fca1d492a66760c3de23f9b4b5cd1af7f2798fe51248c7683ce6fdf2e6ef923" gracePeriod=2
Sep 17 11:17:01.830469 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:17:01.830438    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9048-38c11b\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="cc4a483c-c6e9-4588-8971-a53675b15da1" pod="opendatahub/test-curl-9048-38c11b"
Sep 17 11:17:03.261324 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:03.261292    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:04.011471 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:04.011447    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9048-38c11b"
Sep 17 11:17:04.145692 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:04.145604    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42dvz\" (UniqueName: \"kubernetes.io/projected/cc4a483c-c6e9-4588-8971-a53675b15da1-kube-api-access-42dvz\") pod \"cc4a483c-c6e9-4588-8971-a53675b15da1\" (UID: \"cc4a483c-c6e9-4588-8971-a53675b15da1\") "
Sep 17 11:17:04.147657 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:04.147621    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4a483c-c6e9-4588-8971-a53675b15da1-kube-api-access-42dvz" (OuterVolumeSpecName: "kube-api-access-42dvz") pod "cc4a483c-c6e9-4588-8971-a53675b15da1" (UID: "cc4a483c-c6e9-4588-8971-a53675b15da1"). InnerVolumeSpecName "kube-api-access-42dvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:17:04.246984 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:04.246945    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42dvz\" (UniqueName: \"kubernetes.io/projected/cc4a483c-c6e9-4588-8971-a53675b15da1-kube-api-access-42dvz\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:17:04.455685 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:04.455607    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc4a483c-c6e9-4588-8971-a53675b15da1" path="/var/lib/kubelet/pods/cc4a483c-c6e9-4588-8971-a53675b15da1/volumes"
Sep 17 11:17:04.843946 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:04.843912    2562 generic.go:356] "Generic (PLEG): container finished" podID="cc4a483c-c6e9-4588-8971-a53675b15da1" containerID="2fca1d492a66760c3de23f9b4b5cd1af7f2798fe51248c7683ce6fdf2e6ef923" exitCode=137
Sep 17 11:17:04.844103 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:04.843954    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9048-38c11b"
Sep 17 11:17:04.844103 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:04.843967    2562 scope.go:117] "RemoveContainer" containerID="2fca1d492a66760c3de23f9b4b5cd1af7f2798fe51248c7683ce6fdf2e6ef923"
Sep 17 11:17:04.852522 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:04.852500    2562 scope.go:117] "RemoveContainer" containerID="2fca1d492a66760c3de23f9b4b5cd1af7f2798fe51248c7683ce6fdf2e6ef923"
Sep 17 11:17:04.852855 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:17:04.852833    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fca1d492a66760c3de23f9b4b5cd1af7f2798fe51248c7683ce6fdf2e6ef923\": container with ID starting with 2fca1d492a66760c3de23f9b4b5cd1af7f2798fe51248c7683ce6fdf2e6ef923 not found: ID does not exist" containerID="2fca1d492a66760c3de23f9b4b5cd1af7f2798fe51248c7683ce6fdf2e6ef923"
Sep 17 11:17:04.852918 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:04.852865    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fca1d492a66760c3de23f9b4b5cd1af7f2798fe51248c7683ce6fdf2e6ef923"} err="failed to get container status \"2fca1d492a66760c3de23f9b4b5cd1af7f2798fe51248c7683ce6fdf2e6ef923\": rpc error: code = NotFound desc = could not find container \"2fca1d492a66760c3de23f9b4b5cd1af7f2798fe51248c7683ce6fdf2e6ef923\": container with ID starting with 2fca1d492a66760c3de23f9b4b5cd1af7f2798fe51248c7683ce6fdf2e6ef923 not found: ID does not exist"
Sep 17 11:17:07.160063 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:07.160019    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:17:07.160470 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:07.160089    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:17:07.172743 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:07.172711    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:17:07.875697 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:07.875669    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:17:08.091271 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:08.091241    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"]
Sep 17 11:17:08.125869 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:08.125742    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:09.827652 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:09.827617    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:12.414318 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:12.414287    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:12.493974 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:12.493939    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:15.441729 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:15.441694    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:17:16.561840 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:16.561791    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:23.110384 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.110281    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz" podUID="ec21cb8a-a490-43bc-adb6-d4449b5cc177" containerName="main" containerID="cri-o://001e1b00f8e5c8f808874cf351e60a42436200a60d0819de99df9713b01bd707" gracePeriod=45
Sep 17 11:17:23.302991 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.302957    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:23.361664 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.361607    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:17:23.425167 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.425131    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-dshm\") pod \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") "
Sep 17 11:17:23.425329 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.425176    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-model-cache\") pod \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") "
Sep 17 11:17:23.425329 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.425241    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec21cb8a-a490-43bc-adb6-d4449b5cc177-tls-certs\") pod \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") "
Sep 17 11:17:23.425329 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.425274    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvs27\" (UniqueName: \"kubernetes.io/projected/ec21cb8a-a490-43bc-adb6-d4449b5cc177-kube-api-access-rvs27\") pod \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") "
Sep 17 11:17:23.425329 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.425293    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-home\") pod \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") "
Sep 17 11:17:23.425329 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.425322    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-tmp-dir\") pod \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\" (UID: \"ec21cb8a-a490-43bc-adb6-d4449b5cc177\") "
Sep 17 11:17:23.425655 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.425527    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec21cb8a-a490-43bc-adb6-d4449b5cc177" (UID: "ec21cb8a-a490-43bc-adb6-d4449b5cc177"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:17:23.425655 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.425567    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-home" (OuterVolumeSpecName: "home") pod "ec21cb8a-a490-43bc-adb6-d4449b5cc177" (UID: "ec21cb8a-a490-43bc-adb6-d4449b5cc177"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:17:23.425655 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.425601    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec21cb8a-a490-43bc-adb6-d4449b5cc177" (UID: "ec21cb8a-a490-43bc-adb6-d4449b5cc177"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:17:23.427457 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.427429    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec21cb8a-a490-43bc-adb6-d4449b5cc177-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec21cb8a-a490-43bc-adb6-d4449b5cc177" (UID: "ec21cb8a-a490-43bc-adb6-d4449b5cc177"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:17:23.427572 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.427455    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec21cb8a-a490-43bc-adb6-d4449b5cc177-kube-api-access-rvs27" (OuterVolumeSpecName: "kube-api-access-rvs27") pod "ec21cb8a-a490-43bc-adb6-d4449b5cc177" (UID: "ec21cb8a-a490-43bc-adb6-d4449b5cc177"). InnerVolumeSpecName "kube-api-access-rvs27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:17:23.427572 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.427490    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-dshm" (OuterVolumeSpecName: "dshm") pod "ec21cb8a-a490-43bc-adb6-d4449b5cc177" (UID: "ec21cb8a-a490-43bc-adb6-d4449b5cc177"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:17:23.526743 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.526700    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec21cb8a-a490-43bc-adb6-d4449b5cc177-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:17:23.526743 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.526735    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvs27\" (UniqueName: \"kubernetes.io/projected/ec21cb8a-a490-43bc-adb6-d4449b5cc177-kube-api-access-rvs27\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:17:23.526743 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.526744    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:17:23.527009 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.526774    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:17:23.527009 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.526783    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:17:23.527009 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.526791    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec21cb8a-a490-43bc-adb6-d4449b5cc177-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:17:23.948498 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.948463    2562 generic.go:356] "Generic (PLEG): container finished" podID="ec21cb8a-a490-43bc-adb6-d4449b5cc177" containerID="001e1b00f8e5c8f808874cf351e60a42436200a60d0819de99df9713b01bd707" exitCode=0
Sep 17 11:17:23.948657 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.948520    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"
Sep 17 11:17:23.948657 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.948534    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz" event={"ID":"ec21cb8a-a490-43bc-adb6-d4449b5cc177","Type":"ContainerDied","Data":"001e1b00f8e5c8f808874cf351e60a42436200a60d0819de99df9713b01bd707"}
Sep 17 11:17:23.948657 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.948569    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz" event={"ID":"ec21cb8a-a490-43bc-adb6-d4449b5cc177","Type":"ContainerDied","Data":"e6228b521e85eab5a50d1abb5ad577ee2660d7e8308c647f134df472f876b96c"}
Sep 17 11:17:23.948657 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.948585    2562 scope.go:117] "RemoveContainer" containerID="001e1b00f8e5c8f808874cf351e60a42436200a60d0819de99df9713b01bd707"
Sep 17 11:17:23.958123 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.958103    2562 scope.go:117] "RemoveContainer" containerID="001e1b00f8e5c8f808874cf351e60a42436200a60d0819de99df9713b01bd707"
Sep 17 11:17:23.958367 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:17:23.958347    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"001e1b00f8e5c8f808874cf351e60a42436200a60d0819de99df9713b01bd707\": container with ID starting with 001e1b00f8e5c8f808874cf351e60a42436200a60d0819de99df9713b01bd707 not found: ID does not exist" containerID="001e1b00f8e5c8f808874cf351e60a42436200a60d0819de99df9713b01bd707"
Sep 17 11:17:23.958417 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.958376    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"001e1b00f8e5c8f808874cf351e60a42436200a60d0819de99df9713b01bd707"} err="failed to get container status \"001e1b00f8e5c8f808874cf351e60a42436200a60d0819de99df9713b01bd707\": rpc error: code = NotFound desc = could not find container \"001e1b00f8e5c8f808874cf351e60a42436200a60d0819de99df9713b01bd707\": container with ID starting with 001e1b00f8e5c8f808874cf351e60a42436200a60d0819de99df9713b01bd707 not found: ID does not exist"
Sep 17 11:17:23.971062 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.971037    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"]
Sep 17 11:17:23.975018 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:23.974988    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-026f86/auth-test-model-026f86-kserve-6c945fb76c-7cvhz"]
Sep 17 11:17:24.064458 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:24.064427    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:24.456512 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:24.456479    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec21cb8a-a490-43bc-adb6-d4449b5cc177" path="/var/lib/kubelet/pods/ec21cb8a-a490-43bc-adb6-d4449b5cc177/volumes"
Sep 17 11:17:24.775958 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:24.775885    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:25.641449 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:25.641391    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:26.019254 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.019175    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"]
Sep 17 11:17:26.019580 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.019567    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec21cb8a-a490-43bc-adb6-d4449b5cc177" containerName="main"
Sep 17 11:17:26.019622 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.019582    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec21cb8a-a490-43bc-adb6-d4449b5cc177" containerName="main"
Sep 17 11:17:26.019622 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.019598    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc4a483c-c6e9-4588-8971-a53675b15da1" containerName="test-curl-9048-38c11b"
Sep 17 11:17:26.019622 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.019605    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc4a483c-c6e9-4588-8971-a53675b15da1" containerName="test-curl-9048-38c11b"
Sep 17 11:17:26.019720 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.019676    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc4a483c-c6e9-4588-8971-a53675b15da1" containerName="test-curl-9048-38c11b"
Sep 17 11:17:26.019720 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.019687    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec21cb8a-a490-43bc-adb6-d4449b5cc177" containerName="main"
Sep 17 11:17:26.024104 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.024087    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.028261 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.028240    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-026f86\"/\"sub-test-model-026f86-kserve-self-signed-certs\""
Sep 17 11:17:26.028373 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.028321    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-026f86\"/\"default-dockercfg-j5nx6\""
Sep 17 11:17:26.028506 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.028368    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-026f86\"/\"openshift-service-ca.crt\""
Sep 17 11:17:26.028506 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.028457    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-026f86\"/\"kube-root-ca.crt\""
Sep 17 11:17:26.033317 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.033294    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"]
Sep 17 11:17:26.049164 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.049134    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqbtk\" (UniqueName: \"kubernetes.io/projected/2a67cda4-e481-493e-9276-466ab4e2aa45-kube-api-access-dqbtk\") pod \"sub-test-model-026f86-kserve-56df7cf975-g2gr7\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") " pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.049323 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.049173    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-home\") pod \"sub-test-model-026f86-kserve-56df7cf975-g2gr7\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") " pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.049323 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.049216    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a67cda4-e481-493e-9276-466ab4e2aa45-tls-certs\") pod \"sub-test-model-026f86-kserve-56df7cf975-g2gr7\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") " pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.049323 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.049294    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-tmp-dir\") pod \"sub-test-model-026f86-kserve-56df7cf975-g2gr7\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") " pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.049492 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.049379    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-model-cache\") pod \"sub-test-model-026f86-kserve-56df7cf975-g2gr7\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") " pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.049492 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.049424    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-dshm\") pod \"sub-test-model-026f86-kserve-56df7cf975-g2gr7\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") " pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.119007 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.118969    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:26.149992 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.149964    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-tmp-dir\") pod \"sub-test-model-026f86-kserve-56df7cf975-g2gr7\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") " pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.150144 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.150013    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-model-cache\") pod \"sub-test-model-026f86-kserve-56df7cf975-g2gr7\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") " pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.150144 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.150103    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-dshm\") pod \"sub-test-model-026f86-kserve-56df7cf975-g2gr7\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") " pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.150144 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.150135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqbtk\" (UniqueName: \"kubernetes.io/projected/2a67cda4-e481-493e-9276-466ab4e2aa45-kube-api-access-dqbtk\") pod \"sub-test-model-026f86-kserve-56df7cf975-g2gr7\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") " pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.150288 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.150152    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-home\") pod \"sub-test-model-026f86-kserve-56df7cf975-g2gr7\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") " pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.150288 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.150182    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a67cda4-e481-493e-9276-466ab4e2aa45-tls-certs\") pod \"sub-test-model-026f86-kserve-56df7cf975-g2gr7\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") " pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.150411 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.150385    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-tmp-dir\") pod \"sub-test-model-026f86-kserve-56df7cf975-g2gr7\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") " pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.150473 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.150451    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-model-cache\") pod \"sub-test-model-026f86-kserve-56df7cf975-g2gr7\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") " pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.150533 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.150505    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-home\") pod \"sub-test-model-026f86-kserve-56df7cf975-g2gr7\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") " pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.152357 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.152334    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-dshm\") pod \"sub-test-model-026f86-kserve-56df7cf975-g2gr7\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") " pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.152749 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.152731    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a67cda4-e481-493e-9276-466ab4e2aa45-tls-certs\") pod \"sub-test-model-026f86-kserve-56df7cf975-g2gr7\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") " pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.158785 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.158762    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqbtk\" (UniqueName: \"kubernetes.io/projected/2a67cda4-e481-493e-9276-466ab4e2aa45-kube-api-access-dqbtk\") pod \"sub-test-model-026f86-kserve-56df7cf975-g2gr7\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") " pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.336896 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.336870    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-026f86\"/\"default-dockercfg-j5nx6\""
Sep 17 11:17:26.343942 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.343919    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:26.573076 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.573044    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:26.626420 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.626394    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"]
Sep 17 11:17:26.627290 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:17:26.627265    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a67cda4_e481_493e_9276_466ab4e2aa45.slice/crio-78f6bb2423aed68d4c66ac41a02a835559aaeacfe7ee355803874518f0fe305b WatchSource:0}: Error finding container 78f6bb2423aed68d4c66ac41a02a835559aaeacfe7ee355803874518f0fe305b: Status 404 returned error can't find the container with id 78f6bb2423aed68d4c66ac41a02a835559aaeacfe7ee355803874518f0fe305b
Sep 17 11:17:26.966954 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.966867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7" event={"ID":"2a67cda4-e481-493e-9276-466ab4e2aa45","Type":"ContainerStarted","Data":"849ee94fec7c5d5cf796fb5f51076576996030826e62522b4c176e45be654e3e"}
Sep 17 11:17:26.966954 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.966917    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7" event={"ID":"2a67cda4-e481-493e-9276-466ab4e2aa45","Type":"ContainerStarted","Data":"78f6bb2423aed68d4c66ac41a02a835559aaeacfe7ee355803874518f0fe305b"}
Sep 17 11:17:26.986972 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:26.986928    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7" podStartSLOduration=0.986914178 podStartE2EDuration="986.914178ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:17:26.984799806 +0000 UTC m=+1441.082845375" watchObservedRunningTime="2026-09-17 11:17:26.986914178 +0000 UTC m=+1441.084959743"
Sep 17 11:17:27.438935 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:27.438906    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:30.315748 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:30.315712    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:17:36.344721 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:36.344673    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:36.344721 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:36.344731    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:36.357125 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:36.357099    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:37.030461 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:37.030433    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:17:38.739836 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:38.739787    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"]
Sep 17 11:17:38.740259 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:38.739999    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8" podUID="982b56f5-c2b6-4f78-bc8a-ce51d455c631" containerName="maas-api" containerID="cri-o://a5245fe408e1f7d705fbbe7f6c23041c80cd25c3187a1f55d5569f0a447851e8" gracePeriod=30
Sep 17 11:17:38.763157 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:38.763122    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:38.860791 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:38.860765    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:38.961353 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:38.961307    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:38.990082 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:38.990022    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:17:39.030512 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.030482    2562 generic.go:356] "Generic (PLEG): container finished" podID="982b56f5-c2b6-4f78-bc8a-ce51d455c631" containerID="a5245fe408e1f7d705fbbe7f6c23041c80cd25c3187a1f55d5569f0a447851e8" exitCode=0
Sep 17 11:17:39.030653 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.030531    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"
Sep 17 11:17:39.030653 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.030562    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8" event={"ID":"982b56f5-c2b6-4f78-bc8a-ce51d455c631","Type":"ContainerDied","Data":"a5245fe408e1f7d705fbbe7f6c23041c80cd25c3187a1f55d5569f0a447851e8"}
Sep 17 11:17:39.030653 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.030620    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8" event={"ID":"982b56f5-c2b6-4f78-bc8a-ce51d455c631","Type":"ContainerDied","Data":"e7d75634b0d6b701edcc9439d694ef878bb19128259b78442d7b0b2707acf4c8"}
Sep 17 11:17:39.030775 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.030653    2562 scope.go:117] "RemoveContainer" containerID="a5245fe408e1f7d705fbbe7f6c23041c80cd25c3187a1f55d5569f0a447851e8"
Sep 17 11:17:39.039723 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.039697    2562 scope.go:117] "RemoveContainer" containerID="a5245fe408e1f7d705fbbe7f6c23041c80cd25c3187a1f55d5569f0a447851e8"
Sep 17 11:17:39.040070 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:17:39.040046    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5245fe408e1f7d705fbbe7f6c23041c80cd25c3187a1f55d5569f0a447851e8\": container with ID starting with a5245fe408e1f7d705fbbe7f6c23041c80cd25c3187a1f55d5569f0a447851e8 not found: ID does not exist" containerID="a5245fe408e1f7d705fbbe7f6c23041c80cd25c3187a1f55d5569f0a447851e8"
Sep 17 11:17:39.040196 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.040078    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5245fe408e1f7d705fbbe7f6c23041c80cd25c3187a1f55d5569f0a447851e8"} err="failed to get container status \"a5245fe408e1f7d705fbbe7f6c23041c80cd25c3187a1f55d5569f0a447851e8\": rpc error: code = NotFound desc = could not find container \"a5245fe408e1f7d705fbbe7f6c23041c80cd25c3187a1f55d5569f0a447851e8\": container with ID starting with a5245fe408e1f7d705fbbe7f6c23041c80cd25c3187a1f55d5569f0a447851e8 not found: ID does not exist"
Sep 17 11:17:39.061303 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.061277    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:39.079422 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.079396    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54vvt\" (UniqueName: \"kubernetes.io/projected/982b56f5-c2b6-4f78-bc8a-ce51d455c631-kube-api-access-54vvt\") pod \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\" (UID: \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\") "
Sep 17 11:17:39.079549 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.079470    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/982b56f5-c2b6-4f78-bc8a-ce51d455c631-openshift-service-ca\") pod \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\" (UID: \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\") "
Sep 17 11:17:39.079549 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.079501    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982b56f5-c2b6-4f78-bc8a-ce51d455c631-metrics-certs\") pod \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\" (UID: \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\") "
Sep 17 11:17:39.079549 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.079531    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982b56f5-c2b6-4f78-bc8a-ce51d455c631-maas-api-tls\") pod \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\" (UID: \"982b56f5-c2b6-4f78-bc8a-ce51d455c631\") "
Sep 17 11:17:39.079928 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.079903    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/982b56f5-c2b6-4f78-bc8a-ce51d455c631-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "982b56f5-c2b6-4f78-bc8a-ce51d455c631" (UID: "982b56f5-c2b6-4f78-bc8a-ce51d455c631"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:17:39.081606 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.081579    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/982b56f5-c2b6-4f78-bc8a-ce51d455c631-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "982b56f5-c2b6-4f78-bc8a-ce51d455c631" (UID: "982b56f5-c2b6-4f78-bc8a-ce51d455c631"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:17:39.082034 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.082019    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/982b56f5-c2b6-4f78-bc8a-ce51d455c631-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "982b56f5-c2b6-4f78-bc8a-ce51d455c631" (UID: "982b56f5-c2b6-4f78-bc8a-ce51d455c631"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:17:39.082108 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.082056    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982b56f5-c2b6-4f78-bc8a-ce51d455c631-kube-api-access-54vvt" (OuterVolumeSpecName: "kube-api-access-54vvt") pod "982b56f5-c2b6-4f78-bc8a-ce51d455c631" (UID: "982b56f5-c2b6-4f78-bc8a-ce51d455c631"). InnerVolumeSpecName "kube-api-access-54vvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:17:39.181112 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.181073    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/982b56f5-c2b6-4f78-bc8a-ce51d455c631-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:17:39.181112 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.181103    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982b56f5-c2b6-4f78-bc8a-ce51d455c631-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:17:39.181112 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.181112    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982b56f5-c2b6-4f78-bc8a-ce51d455c631-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:17:39.181112 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.181120    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54vvt\" (UniqueName: \"kubernetes.io/projected/982b56f5-c2b6-4f78-bc8a-ce51d455c631-kube-api-access-54vvt\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:17:39.353052 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.353020    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"]
Sep 17 11:17:39.357256 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.357232    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d727624f-5f9988845f-hgvv8"]
Sep 17 11:17:39.431385 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:39.431354    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:40.227182 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:40.227151    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:40.456395 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:40.456361    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="982b56f5-c2b6-4f78-bc8a-ce51d455c631" path="/var/lib/kubelet/pods/982b56f5-c2b6-4f78-bc8a-ce51d455c631/volumes"
Sep 17 11:17:41.523495 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:41.523464    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:42.245898 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:42.245869    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:42.346926 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:42.346895    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:47.008490 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:47.008452    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:17:47.239301 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:47.239268    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"]
Sep 17 11:17:47.351259 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:47.351225    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:54.611312 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.611277    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"]
Sep 17 11:17:54.611885 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.611866    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="982b56f5-c2b6-4f78-bc8a-ce51d455c631" containerName="maas-api"
Sep 17 11:17:54.611962 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.611887    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="982b56f5-c2b6-4f78-bc8a-ce51d455c631" containerName="maas-api"
Sep 17 11:17:54.612030 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.612012    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="982b56f5-c2b6-4f78-bc8a-ce51d455c631" containerName="maas-api"
Sep 17 11:17:54.616425 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.616404    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.620951 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.620906    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8287f\"/\"openshift-service-ca.crt\""
Sep 17 11:17:54.620951 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.620928    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8287f\"/\"default-dockercfg-jpfb4\""
Sep 17 11:17:54.620951 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.620931    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8287f\"/\"kube-root-ca.crt\""
Sep 17 11:17:54.621182 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.620906    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8287f\"/\"sub-test-model-b8287f-kserve-self-signed-certs\""
Sep 17 11:17:54.626034 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.626009    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"]
Sep 17 11:17:54.635649 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.635628    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-home\") pod \"sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.635759 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.635654    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-model-cache\") pod \"sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.635759 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.635682    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-dshm\") pod \"sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.635759 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.635721    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-tls-certs\") pod \"sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.635759 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.635747    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-tmp-dir\") pod \"sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.635954 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.635807    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6fnv\" (UniqueName: \"kubernetes.io/projected/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-kube-api-access-t6fnv\") pod \"sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.652287 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.652266    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:17:54.737165 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.737128    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-dshm\") pod \"sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.737343 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.737186    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-tls-certs\") pod \"sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.737343 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.737220    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-tmp-dir\") pod \"sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.737343 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.737273    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6fnv\" (UniqueName: \"kubernetes.io/projected/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-kube-api-access-t6fnv\") pod \"sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.737343 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.737334    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-home\") pod \"sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.737581 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.737466    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-model-cache\") pod \"sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.737679 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.737646    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-tmp-dir\") pod \"sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.737734 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.737696    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-home\") pod \"sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.737796 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.737778    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-model-cache\") pod \"sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.739491 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.739466    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-dshm\") pod \"sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.739754 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.739737    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-tls-certs\") pod \"sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.746211 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.746193    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6fnv\" (UniqueName: \"kubernetes.io/projected/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-kube-api-access-t6fnv\") pod \"sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:54.926870 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:54.926748    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:17:55.198753 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:55.198710    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"]
Sep 17 11:17:55.201105 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:17:55.201079    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59b421d6_b8ce_4b1a_b6fe_98d20789ae46.slice/crio-c273a63144f6e7db00c871f5badb499fba5a864a36b4ab489849e27c00506f2e WatchSource:0}: Error finding container c273a63144f6e7db00c871f5badb499fba5a864a36b4ab489849e27c00506f2e: Status 404 returned error can't find the container with id c273a63144f6e7db00c871f5badb499fba5a864a36b4ab489849e27c00506f2e
Sep 17 11:17:56.128485 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:56.128453    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd" event={"ID":"59b421d6-b8ce-4b1a-b6fe-98d20789ae46","Type":"ContainerStarted","Data":"cff834d1ab87c98a7e54aeaaba155f00000d57692b3aa8277a34c8619d987ff6"}
Sep 17 11:17:56.128485 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:56.128492    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd" event={"ID":"59b421d6-b8ce-4b1a-b6fe-98d20789ae46","Type":"ContainerStarted","Data":"c273a63144f6e7db00c871f5badb499fba5a864a36b4ab489849e27c00506f2e"}
Sep 17 11:17:56.146911 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:56.146863    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd" podStartSLOduration=2.146849093 podStartE2EDuration="2.146849093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:17:56.146009783 +0000 UTC m=+1470.244055363" watchObservedRunningTime="2026-09-17 11:17:56.146849093 +0000 UTC m=+1470.244894658"
Sep 17 11:17:59.124598 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:17:59.124555    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:18:02.258560 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.258495    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq" podUID="b51be080-dc53-4c7f-aa57-edc475c5679e" containerName="main" containerID="cri-o://bc7056c8656df707dbe45b7651159534e171349d47d48e7add4d47b5e3dd4112" gracePeriod=45
Sep 17 11:18:02.504237 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.504210    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:18:02.613932 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.613888    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-tmp-dir\") pod \"b51be080-dc53-4c7f-aa57-edc475c5679e\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") "
Sep 17 11:18:02.613932 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.613939    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b51be080-dc53-4c7f-aa57-edc475c5679e-tls-certs\") pod \"b51be080-dc53-4c7f-aa57-edc475c5679e\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") "
Sep 17 11:18:02.614214 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.613961    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-home\") pod \"b51be080-dc53-4c7f-aa57-edc475c5679e\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") "
Sep 17 11:18:02.614214 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.614026    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-model-cache\") pod \"b51be080-dc53-4c7f-aa57-edc475c5679e\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") "
Sep 17 11:18:02.614214 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.614064    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-dshm\") pod \"b51be080-dc53-4c7f-aa57-edc475c5679e\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") "
Sep 17 11:18:02.614214 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.614104    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npvmp\" (UniqueName: \"kubernetes.io/projected/b51be080-dc53-4c7f-aa57-edc475c5679e-kube-api-access-npvmp\") pod \"b51be080-dc53-4c7f-aa57-edc475c5679e\" (UID: \"b51be080-dc53-4c7f-aa57-edc475c5679e\") "
Sep 17 11:18:02.614402 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.614242    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b51be080-dc53-4c7f-aa57-edc475c5679e" (UID: "b51be080-dc53-4c7f-aa57-edc475c5679e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:18:02.614402 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.614281    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-home" (OuterVolumeSpecName: "home") pod "b51be080-dc53-4c7f-aa57-edc475c5679e" (UID: "b51be080-dc53-4c7f-aa57-edc475c5679e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:18:02.614402 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.614320    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b51be080-dc53-4c7f-aa57-edc475c5679e" (UID: "b51be080-dc53-4c7f-aa57-edc475c5679e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:18:02.614542 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.614510    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:02.614542 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.614524    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:02.614542 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.614533    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:02.616197 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.616176    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b51be080-dc53-4c7f-aa57-edc475c5679e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b51be080-dc53-4c7f-aa57-edc475c5679e" (UID: "b51be080-dc53-4c7f-aa57-edc475c5679e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:18:02.616278 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.616257    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51be080-dc53-4c7f-aa57-edc475c5679e-kube-api-access-npvmp" (OuterVolumeSpecName: "kube-api-access-npvmp") pod "b51be080-dc53-4c7f-aa57-edc475c5679e" (UID: "b51be080-dc53-4c7f-aa57-edc475c5679e"). InnerVolumeSpecName "kube-api-access-npvmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:18:02.616363 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.616346    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-dshm" (OuterVolumeSpecName: "dshm") pod "b51be080-dc53-4c7f-aa57-edc475c5679e" (UID: "b51be080-dc53-4c7f-aa57-edc475c5679e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:18:02.716093 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.716042    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b51be080-dc53-4c7f-aa57-edc475c5679e-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:02.716093 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.716087    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b51be080-dc53-4c7f-aa57-edc475c5679e-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:02.716093 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:02.716097    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npvmp\" (UniqueName: \"kubernetes.io/projected/b51be080-dc53-4c7f-aa57-edc475c5679e-kube-api-access-npvmp\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:03.166442 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:03.166406    2562 generic.go:356] "Generic (PLEG): container finished" podID="b51be080-dc53-4c7f-aa57-edc475c5679e" containerID="bc7056c8656df707dbe45b7651159534e171349d47d48e7add4d47b5e3dd4112" exitCode=0
Sep 17 11:18:03.166644 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:03.166453    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"
Sep 17 11:18:03.166644 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:03.166503    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq" event={"ID":"b51be080-dc53-4c7f-aa57-edc475c5679e","Type":"ContainerDied","Data":"bc7056c8656df707dbe45b7651159534e171349d47d48e7add4d47b5e3dd4112"}
Sep 17 11:18:03.166644 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:03.166552    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq" event={"ID":"b51be080-dc53-4c7f-aa57-edc475c5679e","Type":"ContainerDied","Data":"c3e4e7b70b144883f68bbc0fdd76034d8c73a2eaf9d97ea99287cf35dfd850a5"}
Sep 17 11:18:03.166644 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:03.166573    2562 scope.go:117] "RemoveContainer" containerID="bc7056c8656df707dbe45b7651159534e171349d47d48e7add4d47b5e3dd4112"
Sep 17 11:18:03.175887 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:03.175867    2562 scope.go:117] "RemoveContainer" containerID="bc7056c8656df707dbe45b7651159534e171349d47d48e7add4d47b5e3dd4112"
Sep 17 11:18:03.176180 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:18:03.176155    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc7056c8656df707dbe45b7651159534e171349d47d48e7add4d47b5e3dd4112\": container with ID starting with bc7056c8656df707dbe45b7651159534e171349d47d48e7add4d47b5e3dd4112 not found: ID does not exist" containerID="bc7056c8656df707dbe45b7651159534e171349d47d48e7add4d47b5e3dd4112"
Sep 17 11:18:03.176277 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:03.176185    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc7056c8656df707dbe45b7651159534e171349d47d48e7add4d47b5e3dd4112"} err="failed to get container status \"bc7056c8656df707dbe45b7651159534e171349d47d48e7add4d47b5e3dd4112\": rpc error: code = NotFound desc = could not find container \"bc7056c8656df707dbe45b7651159534e171349d47d48e7add4d47b5e3dd4112\": container with ID starting with bc7056c8656df707dbe45b7651159534e171349d47d48e7add4d47b5e3dd4112 not found: ID does not exist"
Sep 17 11:18:03.190859 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:03.190809    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"]
Sep 17 11:18:03.196373 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:03.196351    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d727624f/e2e-lifecycle-model-d727624f-kserve-69bf5d8979-htvbq"]
Sep 17 11:18:04.458841 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:04.457705    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b51be080-dc53-4c7f-aa57-edc475c5679e" path="/var/lib/kubelet/pods/b51be080-dc53-4c7f-aa57-edc475c5679e/volumes"
Sep 17 11:18:04.927501 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:04.927461    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:18:04.927717 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:04.927515    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:18:04.940667 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:04.940641    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:18:05.189541 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:05.189453    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:18:16.286225 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:16.286189    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:18:16.386865 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:16.386799    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:18:23.818210 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:23.818170    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:18:25.212386 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:25.212353    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:18:25.323124 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:25.323092    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:18:26.485653 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:26.485625    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 11:18:26.487955 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:26.487936    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 11:18:31.055116 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:31.055050    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:18:31.155981 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:31.155907    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:18:31.915196 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:31.915156    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:18:33.571032 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:33.570996    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:18:33.778318 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:33.778280    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:18:35.423347 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:35.423307    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:18:37.608275 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:37.608234    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9048-19d28f"]
Sep 17 11:18:37.608841 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:37.608799    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b51be080-dc53-4c7f-aa57-edc475c5679e" containerName="main"
Sep 17 11:18:37.608841 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:37.608836    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b51be080-dc53-4c7f-aa57-edc475c5679e" containerName="main"
Sep 17 11:18:37.608995 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:37.608981    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b51be080-dc53-4c7f-aa57-edc475c5679e" containerName="main"
Sep 17 11:18:37.612188 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:37.612168    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9048-19d28f"
Sep 17 11:18:37.615122 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:37.615102    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bnkhs\""
Sep 17 11:18:37.618929 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:37.618904    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9048-19d28f"]
Sep 17 11:18:37.647762 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:37.647728    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4hnf\" (UniqueName: \"kubernetes.io/projected/1cf3d718-0e3d-4b27-ba53-6dead2147acb-kube-api-access-g4hnf\") pod \"test-curl-9048-19d28f\" (UID: \"1cf3d718-0e3d-4b27-ba53-6dead2147acb\") " pod="opendatahub/test-curl-9048-19d28f"
Sep 17 11:18:37.748805 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:37.748767    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4hnf\" (UniqueName: \"kubernetes.io/projected/1cf3d718-0e3d-4b27-ba53-6dead2147acb-kube-api-access-g4hnf\") pod \"test-curl-9048-19d28f\" (UID: \"1cf3d718-0e3d-4b27-ba53-6dead2147acb\") " pod="opendatahub/test-curl-9048-19d28f"
Sep 17 11:18:37.758625 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:37.758593    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4hnf\" (UniqueName: \"kubernetes.io/projected/1cf3d718-0e3d-4b27-ba53-6dead2147acb-kube-api-access-g4hnf\") pod \"test-curl-9048-19d28f\" (UID: \"1cf3d718-0e3d-4b27-ba53-6dead2147acb\") " pod="opendatahub/test-curl-9048-19d28f"
Sep 17 11:18:37.922511 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:37.922393    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9048-19d28f"
Sep 17 11:18:38.191372 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:38.191342    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9048-19d28f"]
Sep 17 11:18:38.194494 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:18:38.194463    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cf3d718_0e3d_4b27_ba53_6dead2147acb.slice/crio-33ae529989c4e5222373040d698dca44dc4b0e4349867a3726070546e5bdf8c2 WatchSource:0}: Error finding container 33ae529989c4e5222373040d698dca44dc4b0e4349867a3726070546e5bdf8c2: Status 404 returned error can't find the container with id 33ae529989c4e5222373040d698dca44dc4b0e4349867a3726070546e5bdf8c2
Sep 17 11:18:38.347265 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:38.347229    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9048-19d28f" event={"ID":"1cf3d718-0e3d-4b27-ba53-6dead2147acb","Type":"ContainerStarted","Data":"efe2a6e182588f3414e027901b3d1687f5bd2cfef37d1ebc61600bcc40cab10b"}
Sep 17 11:18:38.347451 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:38.347280    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9048-19d28f" event={"ID":"1cf3d718-0e3d-4b27-ba53-6dead2147acb","Type":"ContainerStarted","Data":"33ae529989c4e5222373040d698dca44dc4b0e4349867a3726070546e5bdf8c2"}
Sep 17 11:18:38.364350 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:38.364301    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9048-19d28f" podStartSLOduration=1.364285998 podStartE2EDuration="1.364285998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:18:38.363711506 +0000 UTC m=+1512.461757097" watchObservedRunningTime="2026-09-17 11:18:38.364285998 +0000 UTC m=+1512.462331564"
Sep 17 11:18:38.950248 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:38.950218    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9048-19d28f"]
Sep 17 11:18:38.954699 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:38.954671    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9048-19d28f"]
Sep 17 11:18:39.069053 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:39.069019    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9048-01ac74"]
Sep 17 11:18:39.069442 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:39.069429    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cf3d718-0e3d-4b27-ba53-6dead2147acb" containerName="test-curl-9048-19d28f"
Sep 17 11:18:39.069494 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:39.069443    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cf3d718-0e3d-4b27-ba53-6dead2147acb" containerName="test-curl-9048-19d28f"
Sep 17 11:18:39.069535 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:39.069529    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cf3d718-0e3d-4b27-ba53-6dead2147acb" containerName="test-curl-9048-19d28f"
Sep 17 11:18:39.072660 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:39.072644    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9048-01ac74"
Sep 17 11:18:39.079209 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:39.079187    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9048-01ac74"]
Sep 17 11:18:39.160468 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:39.160352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg22r\" (UniqueName: \"kubernetes.io/projected/17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4-kube-api-access-dg22r\") pod \"test-curl-9048-01ac74\" (UID: \"17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4\") " pod="opendatahub/test-curl-9048-01ac74"
Sep 17 11:18:39.261860 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:39.261758    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg22r\" (UniqueName: \"kubernetes.io/projected/17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4-kube-api-access-dg22r\") pod \"test-curl-9048-01ac74\" (UID: \"17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4\") " pod="opendatahub/test-curl-9048-01ac74"
Sep 17 11:18:39.270581 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:39.270556    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg22r\" (UniqueName: \"kubernetes.io/projected/17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4-kube-api-access-dg22r\") pod \"test-curl-9048-01ac74\" (UID: \"17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4\") " pod="opendatahub/test-curl-9048-01ac74"
Sep 17 11:18:39.352558 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:18:39.352523    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9048-19d28f" container="test-curl-9048-19d28f"
Sep 17 11:18:39.382984 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:39.382951    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9048-01ac74"
Sep 17 11:18:39.655986 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:39.655958    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9048-01ac74"]
Sep 17 11:18:39.658041 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:18:39.658018    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17e17c5b_0399_401d_8c8b_ccd5bb7e8cb4.slice/crio-2fd1b3b0f8ff9c1b912904bf520feb458738d3a9b439f916972b16055e76aaa4 WatchSource:0}: Error finding container 2fd1b3b0f8ff9c1b912904bf520feb458738d3a9b439f916972b16055e76aaa4: Status 404 returned error can't find the container with id 2fd1b3b0f8ff9c1b912904bf520feb458738d3a9b439f916972b16055e76aaa4
Sep 17 11:18:40.361289 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:40.361252    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9048-01ac74" event={"ID":"17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4","Type":"ContainerStarted","Data":"80f676f05e4ed0b051c500f0c29b700abb6959d264c1550fe2cb0c66a74ddb95"}
Sep 17 11:18:40.361683 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:40.361297    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9048-01ac74" event={"ID":"17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4","Type":"ContainerStarted","Data":"2fd1b3b0f8ff9c1b912904bf520feb458738d3a9b439f916972b16055e76aaa4"}
Sep 17 11:18:40.361683 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:40.361357    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9048-19d28f" podUID="1cf3d718-0e3d-4b27-ba53-6dead2147acb" containerName="test-curl-9048-19d28f" containerID="cri-o://efe2a6e182588f3414e027901b3d1687f5bd2cfef37d1ebc61600bcc40cab10b" gracePeriod=2
Sep 17 11:18:40.378975 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:40.378930    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9048-01ac74" podStartSLOduration=1.3789151099999999 podStartE2EDuration="1.37891511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:18:40.377220028 +0000 UTC m=+1514.475265595" watchObservedRunningTime="2026-09-17 11:18:40.37891511 +0000 UTC m=+1514.476960697"
Sep 17 11:18:40.379969 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:18:40.379945    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9048-19d28f\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="1cf3d718-0e3d-4b27-ba53-6dead2147acb" pod="opendatahub/test-curl-9048-19d28f"
Sep 17 11:18:40.382375 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:18:40.382352    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9048-19d28f\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="1cf3d718-0e3d-4b27-ba53-6dead2147acb" pod="opendatahub/test-curl-9048-19d28f"
Sep 17 11:18:41.010231 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:41.010139    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9048-01ac74"]
Sep 17 11:18:41.024490 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:41.024451    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9048-01ac74"]
Sep 17 11:18:41.367594 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:18:41.367560    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9048-01ac74" container="test-curl-9048-01ac74"
Sep 17 11:18:41.411789 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:41.411752    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:18:41.519642 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:41.519596    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:18:42.164840 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:42.164781    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:18:42.372766 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:42.372739    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9048-01ac74" podUID="17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4" containerName="test-curl-9048-01ac74" containerID="cri-o://80f676f05e4ed0b051c500f0c29b700abb6959d264c1550fe2cb0c66a74ddb95" gracePeriod=2
Sep 17 11:18:42.375582 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:18:42.375554    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9048-01ac74\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4" pod="opendatahub/test-curl-9048-01ac74"
Sep 17 11:18:42.551983 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:42.551961    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9048-19d28f"
Sep 17 11:18:42.593263 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:42.593232    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4hnf\" (UniqueName: \"kubernetes.io/projected/1cf3d718-0e3d-4b27-ba53-6dead2147acb-kube-api-access-g4hnf\") pod \"1cf3d718-0e3d-4b27-ba53-6dead2147acb\" (UID: \"1cf3d718-0e3d-4b27-ba53-6dead2147acb\") "
Sep 17 11:18:42.595639 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:42.595607    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf3d718-0e3d-4b27-ba53-6dead2147acb-kube-api-access-g4hnf" (OuterVolumeSpecName: "kube-api-access-g4hnf") pod "1cf3d718-0e3d-4b27-ba53-6dead2147acb" (UID: "1cf3d718-0e3d-4b27-ba53-6dead2147acb"). InnerVolumeSpecName "kube-api-access-g4hnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:18:42.694002 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:42.693918    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4hnf\" (UniqueName: \"kubernetes.io/projected/1cf3d718-0e3d-4b27-ba53-6dead2147acb-kube-api-access-g4hnf\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:42.823067 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:42.823037    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:18:43.085504 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:43.085473    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"]
Sep 17 11:18:43.104269 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:43.104234    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:18:43.379340 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:43.379261    2562 generic.go:356] "Generic (PLEG): container finished" podID="1cf3d718-0e3d-4b27-ba53-6dead2147acb" containerID="efe2a6e182588f3414e027901b3d1687f5bd2cfef37d1ebc61600bcc40cab10b" exitCode=137
Sep 17 11:18:43.379340 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:43.379306    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9048-19d28f"
Sep 17 11:18:43.379340 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:43.379332    2562 scope.go:117] "RemoveContainer" containerID="efe2a6e182588f3414e027901b3d1687f5bd2cfef37d1ebc61600bcc40cab10b"
Sep 17 11:18:43.388369 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:43.388350    2562 scope.go:117] "RemoveContainer" containerID="efe2a6e182588f3414e027901b3d1687f5bd2cfef37d1ebc61600bcc40cab10b"
Sep 17 11:18:43.388652 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:18:43.388634    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efe2a6e182588f3414e027901b3d1687f5bd2cfef37d1ebc61600bcc40cab10b\": container with ID starting with efe2a6e182588f3414e027901b3d1687f5bd2cfef37d1ebc61600bcc40cab10b not found: ID does not exist" containerID="efe2a6e182588f3414e027901b3d1687f5bd2cfef37d1ebc61600bcc40cab10b"
Sep 17 11:18:43.388699 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:43.388662    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe2a6e182588f3414e027901b3d1687f5bd2cfef37d1ebc61600bcc40cab10b"} err="failed to get container status \"efe2a6e182588f3414e027901b3d1687f5bd2cfef37d1ebc61600bcc40cab10b\": rpc error: code = NotFound desc = could not find container \"efe2a6e182588f3414e027901b3d1687f5bd2cfef37d1ebc61600bcc40cab10b\": container with ID starting with efe2a6e182588f3414e027901b3d1687f5bd2cfef37d1ebc61600bcc40cab10b not found: ID does not exist"
Sep 17 11:18:43.390366 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:18:43.390343    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9048-19d28f\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="1cf3d718-0e3d-4b27-ba53-6dead2147acb" pod="opendatahub/test-curl-9048-19d28f"
Sep 17 11:18:44.106016 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:44.105980    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"]
Sep 17 11:18:44.126733 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:44.126703    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:18:44.127631 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:18:44.127602    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9048-19d28f\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="1cf3d718-0e3d-4b27-ba53-6dead2147acb" pod="opendatahub/test-curl-9048-19d28f"
Sep 17 11:18:44.459002 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:44.458970    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cf3d718-0e3d-4b27-ba53-6dead2147acb" path="/var/lib/kubelet/pods/1cf3d718-0e3d-4b27-ba53-6dead2147acb/volumes"
Sep 17 11:18:44.518077 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:44.518050    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9048-01ac74"
Sep 17 11:18:44.611415 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:44.611380    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg22r\" (UniqueName: \"kubernetes.io/projected/17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4-kube-api-access-dg22r\") pod \"17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4\" (UID: \"17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4\") "
Sep 17 11:18:44.613467 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:44.613441    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4-kube-api-access-dg22r" (OuterVolumeSpecName: "kube-api-access-dg22r") pod "17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4" (UID: "17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4"). InnerVolumeSpecName "kube-api-access-dg22r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:18:44.712788 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:44.712703    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg22r\" (UniqueName: \"kubernetes.io/projected/17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4-kube-api-access-dg22r\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:45.392866 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:45.392811    2562 generic.go:356] "Generic (PLEG): container finished" podID="17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4" containerID="80f676f05e4ed0b051c500f0c29b700abb6959d264c1550fe2cb0c66a74ddb95" exitCode=137
Sep 17 11:18:45.392866 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:45.392868    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9048-01ac74"
Sep 17 11:18:45.393101 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:45.392928    2562 scope.go:117] "RemoveContainer" containerID="80f676f05e4ed0b051c500f0c29b700abb6959d264c1550fe2cb0c66a74ddb95"
Sep 17 11:18:45.401629 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:45.401611    2562 scope.go:117] "RemoveContainer" containerID="80f676f05e4ed0b051c500f0c29b700abb6959d264c1550fe2cb0c66a74ddb95"
Sep 17 11:18:45.401921 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:18:45.401901    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f676f05e4ed0b051c500f0c29b700abb6959d264c1550fe2cb0c66a74ddb95\": container with ID starting with 80f676f05e4ed0b051c500f0c29b700abb6959d264c1550fe2cb0c66a74ddb95 not found: ID does not exist" containerID="80f676f05e4ed0b051c500f0c29b700abb6959d264c1550fe2cb0c66a74ddb95"
Sep 17 11:18:45.402005 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:45.401932    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f676f05e4ed0b051c500f0c29b700abb6959d264c1550fe2cb0c66a74ddb95"} err="failed to get container status \"80f676f05e4ed0b051c500f0c29b700abb6959d264c1550fe2cb0c66a74ddb95\": rpc error: code = NotFound desc = could not find container \"80f676f05e4ed0b051c500f0c29b700abb6959d264c1550fe2cb0c66a74ddb95\": container with ID starting with 80f676f05e4ed0b051c500f0c29b700abb6959d264c1550fe2cb0c66a74ddb95 not found: ID does not exist"
Sep 17 11:18:45.403736 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:18:45.403713    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9048-01ac74\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4" pod="opendatahub/test-curl-9048-01ac74"
Sep 17 11:18:46.455452 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:18:46.455413    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9048-01ac74\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4" pod="opendatahub/test-curl-9048-01ac74"
Sep 17 11:18:46.456624 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:46.456604    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4" path="/var/lib/kubelet/pods/17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4/volumes"
Sep 17 11:18:49.173004 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:49.172923    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:18:49.285304 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:49.285271    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:18:53.620298 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.620261    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"]
Sep 17 11:18:53.620996 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.620977    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4" containerName="test-curl-9048-01ac74"
Sep 17 11:18:53.621050 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.621001    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4" containerName="test-curl-9048-01ac74"
Sep 17 11:18:53.621125 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.621114    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="17e17c5b-0399-401d-8c8b-ccd5bb7e8cb4" containerName="test-curl-9048-01ac74"
Sep 17 11:18:53.625922 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.625904    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.628931 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.628910    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-026f86\"/\"rate-test-model-026f86-kserve-self-signed-certs\""
Sep 17 11:18:53.632692 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.632670    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"]
Sep 17 11:18:53.687921 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.687881    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:18:53.700561 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.700518    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-dshm\") pod \"rate-test-model-026f86-kserve-f99dddb79-2spr7\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") " pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.700742 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.700586    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-tmp-dir\") pod \"rate-test-model-026f86-kserve-f99dddb79-2spr7\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") " pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.700742 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.700614    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-home\") pod \"rate-test-model-026f86-kserve-f99dddb79-2spr7\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") " pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.700742 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.700631    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwwhs\" (UniqueName: \"kubernetes.io/projected/f28fc50b-e566-4644-b648-bbd99a1436ef-kube-api-access-gwwhs\") pod \"rate-test-model-026f86-kserve-f99dddb79-2spr7\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") " pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.700742 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.700649    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-model-cache\") pod \"rate-test-model-026f86-kserve-f99dddb79-2spr7\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") " pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.701006 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.700776    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f28fc50b-e566-4644-b648-bbd99a1436ef-tls-certs\") pod \"rate-test-model-026f86-kserve-f99dddb79-2spr7\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") " pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.802186 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.802149    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f28fc50b-e566-4644-b648-bbd99a1436ef-tls-certs\") pod \"rate-test-model-026f86-kserve-f99dddb79-2spr7\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") " pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.802384 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.802203    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-dshm\") pod \"rate-test-model-026f86-kserve-f99dddb79-2spr7\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") " pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.802384 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.802238    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-tmp-dir\") pod \"rate-test-model-026f86-kserve-f99dddb79-2spr7\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") " pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.802384 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.802260    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-home\") pod \"rate-test-model-026f86-kserve-f99dddb79-2spr7\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") " pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.802384 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.802278    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwwhs\" (UniqueName: \"kubernetes.io/projected/f28fc50b-e566-4644-b648-bbd99a1436ef-kube-api-access-gwwhs\") pod \"rate-test-model-026f86-kserve-f99dddb79-2spr7\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") " pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.802384 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.802296    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-model-cache\") pod \"rate-test-model-026f86-kserve-f99dddb79-2spr7\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") " pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.802744 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.802717    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-home\") pod \"rate-test-model-026f86-kserve-f99dddb79-2spr7\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") " pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.802810 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.802749    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-model-cache\") pod \"rate-test-model-026f86-kserve-f99dddb79-2spr7\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") " pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.802810 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.802782    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-tmp-dir\") pod \"rate-test-model-026f86-kserve-f99dddb79-2spr7\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") " pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.804437 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.804416    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-dshm\") pod \"rate-test-model-026f86-kserve-f99dddb79-2spr7\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") " pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.804754 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.804738    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f28fc50b-e566-4644-b648-bbd99a1436ef-tls-certs\") pod \"rate-test-model-026f86-kserve-f99dddb79-2spr7\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") " pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.811102 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.811077    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwwhs\" (UniqueName: \"kubernetes.io/projected/f28fc50b-e566-4644-b648-bbd99a1436ef-kube-api-access-gwwhs\") pod \"rate-test-model-026f86-kserve-f99dddb79-2spr7\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") " pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:53.937168 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:53.937082    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:18:54.017720 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:54.017690    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:18:54.212689 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:54.212664    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"]
Sep 17 11:18:54.214679 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:18:54.214650    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf28fc50b_e566_4644_b648_bbd99a1436ef.slice/crio-4042943d734a72ebf1a5d04b97443a1836ff54baa0192407cafc6923f4591807 WatchSource:0}: Error finding container 4042943d734a72ebf1a5d04b97443a1836ff54baa0192407cafc6923f4591807: Status 404 returned error can't find the container with id 4042943d734a72ebf1a5d04b97443a1836ff54baa0192407cafc6923f4591807
Sep 17 11:18:54.439523 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:54.439485    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7" event={"ID":"f28fc50b-e566-4644-b648-bbd99a1436ef","Type":"ContainerStarted","Data":"7194443c246f6de9cefc306e56df0c6dd8b661b5d1a507acb418ba458138d519"}
Sep 17 11:18:54.439680 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:54.439538    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7" event={"ID":"f28fc50b-e566-4644-b648-bbd99a1436ef","Type":"ContainerStarted","Data":"4042943d734a72ebf1a5d04b97443a1836ff54baa0192407cafc6923f4591807"}
Sep 17 11:18:54.459966 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:54.459914    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7" podStartSLOduration=1.459898984 podStartE2EDuration="1.459898984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:18:54.458784685 +0000 UTC m=+1528.556830250" watchObservedRunningTime="2026-09-17 11:18:54.459898984 +0000 UTC m=+1528.557944550"
Sep 17 11:18:58.104314 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.104275    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7" podUID="2a67cda4-e481-493e-9276-466ab4e2aa45" containerName="main" containerID="cri-o://849ee94fec7c5d5cf796fb5f51076576996030826e62522b4c176e45be654e3e" gracePeriod=45
Sep 17 11:18:58.355324 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.355259    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:18:58.449496 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.449467    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a67cda4-e481-493e-9276-466ab4e2aa45-tls-certs\") pod \"2a67cda4-e481-493e-9276-466ab4e2aa45\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") "
Sep 17 11:18:58.449674 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.449556    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-model-cache\") pod \"2a67cda4-e481-493e-9276-466ab4e2aa45\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") "
Sep 17 11:18:58.449674 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.449587    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-tmp-dir\") pod \"2a67cda4-e481-493e-9276-466ab4e2aa45\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") "
Sep 17 11:18:58.449674 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.449643    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqbtk\" (UniqueName: \"kubernetes.io/projected/2a67cda4-e481-493e-9276-466ab4e2aa45-kube-api-access-dqbtk\") pod \"2a67cda4-e481-493e-9276-466ab4e2aa45\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") "
Sep 17 11:18:58.449871 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.449698    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-home\") pod \"2a67cda4-e481-493e-9276-466ab4e2aa45\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") "
Sep 17 11:18:58.449871 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.449719    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-dshm\") pod \"2a67cda4-e481-493e-9276-466ab4e2aa45\" (UID: \"2a67cda4-e481-493e-9276-466ab4e2aa45\") "
Sep 17 11:18:58.449871 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.449806    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a67cda4-e481-493e-9276-466ab4e2aa45" (UID: "2a67cda4-e481-493e-9276-466ab4e2aa45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:18:58.450044 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.449988    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a67cda4-e481-493e-9276-466ab4e2aa45" (UID: "2a67cda4-e481-493e-9276-466ab4e2aa45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:18:58.450106 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.450037    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-home" (OuterVolumeSpecName: "home") pod "2a67cda4-e481-493e-9276-466ab4e2aa45" (UID: "2a67cda4-e481-493e-9276-466ab4e2aa45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:18:58.450207 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.450190    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:58.450282 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.450213    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:58.450282 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.450226    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:58.451683 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.451664    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a67cda4-e481-493e-9276-466ab4e2aa45-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a67cda4-e481-493e-9276-466ab4e2aa45" (UID: "2a67cda4-e481-493e-9276-466ab4e2aa45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:18:58.451798 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.451731    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-dshm" (OuterVolumeSpecName: "dshm") pod "2a67cda4-e481-493e-9276-466ab4e2aa45" (UID: "2a67cda4-e481-493e-9276-466ab4e2aa45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:18:58.452161 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.452141    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a67cda4-e481-493e-9276-466ab4e2aa45-kube-api-access-dqbtk" (OuterVolumeSpecName: "kube-api-access-dqbtk") pod "2a67cda4-e481-493e-9276-466ab4e2aa45" (UID: "2a67cda4-e481-493e-9276-466ab4e2aa45"). InnerVolumeSpecName "kube-api-access-dqbtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:18:58.462306 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.462279    2562 generic.go:356] "Generic (PLEG): container finished" podID="2a67cda4-e481-493e-9276-466ab4e2aa45" containerID="849ee94fec7c5d5cf796fb5f51076576996030826e62522b4c176e45be654e3e" exitCode=0
Sep 17 11:18:58.462435 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.462330    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"
Sep 17 11:18:58.462435 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.462332    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7" event={"ID":"2a67cda4-e481-493e-9276-466ab4e2aa45","Type":"ContainerDied","Data":"849ee94fec7c5d5cf796fb5f51076576996030826e62522b4c176e45be654e3e"}
Sep 17 11:18:58.462554 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.462445    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7" event={"ID":"2a67cda4-e481-493e-9276-466ab4e2aa45","Type":"ContainerDied","Data":"78f6bb2423aed68d4c66ac41a02a835559aaeacfe7ee355803874518f0fe305b"}
Sep 17 11:18:58.462554 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.462468    2562 scope.go:117] "RemoveContainer" containerID="849ee94fec7c5d5cf796fb5f51076576996030826e62522b4c176e45be654e3e"
Sep 17 11:18:58.477291 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.477271    2562 scope.go:117] "RemoveContainer" containerID="849ee94fec7c5d5cf796fb5f51076576996030826e62522b4c176e45be654e3e"
Sep 17 11:18:58.477547 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:18:58.477527    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"849ee94fec7c5d5cf796fb5f51076576996030826e62522b4c176e45be654e3e\": container with ID starting with 849ee94fec7c5d5cf796fb5f51076576996030826e62522b4c176e45be654e3e not found: ID does not exist" containerID="849ee94fec7c5d5cf796fb5f51076576996030826e62522b4c176e45be654e3e"
Sep 17 11:18:58.477602 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.477553    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"849ee94fec7c5d5cf796fb5f51076576996030826e62522b4c176e45be654e3e"} err="failed to get container status \"849ee94fec7c5d5cf796fb5f51076576996030826e62522b4c176e45be654e3e\": rpc error: code = NotFound desc = could not find container \"849ee94fec7c5d5cf796fb5f51076576996030826e62522b4c176e45be654e3e\": container with ID starting with 849ee94fec7c5d5cf796fb5f51076576996030826e62522b4c176e45be654e3e not found: ID does not exist"
Sep 17 11:18:58.488231 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.488209    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"]
Sep 17 11:18:58.491852 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.491832    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-026f86/sub-test-model-026f86-kserve-56df7cf975-g2gr7"]
Sep 17 11:18:58.551641 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.551602    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqbtk\" (UniqueName: \"kubernetes.io/projected/2a67cda4-e481-493e-9276-466ab4e2aa45-kube-api-access-dqbtk\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:58.551641 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.551633    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a67cda4-e481-493e-9276-466ab4e2aa45-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:58.551641 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:58.551644    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a67cda4-e481-493e-9276-466ab4e2aa45-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:59.126320 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.126276    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd" podUID="59b421d6-b8ce-4b1a-b6fe-98d20789ae46" containerName="main" containerID="cri-o://cff834d1ab87c98a7e54aeaaba155f00000d57692b3aa8277a34c8619d987ff6" gracePeriod=45
Sep 17 11:18:59.376317 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.376278    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:18:59.460933 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.460851    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-model-cache\") pod \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") "
Sep 17 11:18:59.460933 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.460907    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-dshm\") pod \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") "
Sep 17 11:18:59.461174 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.460955    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-home\") pod \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") "
Sep 17 11:18:59.461174 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.460986    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-tmp-dir\") pod \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") "
Sep 17 11:18:59.461174 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.461013    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6fnv\" (UniqueName: \"kubernetes.io/projected/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-kube-api-access-t6fnv\") pod \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") "
Sep 17 11:18:59.461174 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.461077    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-tls-certs\") pod \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\" (UID: \"59b421d6-b8ce-4b1a-b6fe-98d20789ae46\") "
Sep 17 11:18:59.461388 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.461231    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-model-cache" (OuterVolumeSpecName: "model-cache") pod "59b421d6-b8ce-4b1a-b6fe-98d20789ae46" (UID: "59b421d6-b8ce-4b1a-b6fe-98d20789ae46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:18:59.461448 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.461386    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59b421d6-b8ce-4b1a-b6fe-98d20789ae46" (UID: "59b421d6-b8ce-4b1a-b6fe-98d20789ae46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:18:59.461448 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.461404    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:59.461448 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.461444    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-home" (OuterVolumeSpecName: "home") pod "59b421d6-b8ce-4b1a-b6fe-98d20789ae46" (UID: "59b421d6-b8ce-4b1a-b6fe-98d20789ae46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:18:59.463756 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.463716    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59b421d6-b8ce-4b1a-b6fe-98d20789ae46" (UID: "59b421d6-b8ce-4b1a-b6fe-98d20789ae46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:18:59.463920 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.463861    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-kube-api-access-t6fnv" (OuterVolumeSpecName: "kube-api-access-t6fnv") pod "59b421d6-b8ce-4b1a-b6fe-98d20789ae46" (UID: "59b421d6-b8ce-4b1a-b6fe-98d20789ae46"). InnerVolumeSpecName "kube-api-access-t6fnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:18:59.463920 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.463867    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-dshm" (OuterVolumeSpecName: "dshm") pod "59b421d6-b8ce-4b1a-b6fe-98d20789ae46" (UID: "59b421d6-b8ce-4b1a-b6fe-98d20789ae46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:18:59.469252 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.469217    2562 generic.go:356] "Generic (PLEG): container finished" podID="59b421d6-b8ce-4b1a-b6fe-98d20789ae46" containerID="cff834d1ab87c98a7e54aeaaba155f00000d57692b3aa8277a34c8619d987ff6" exitCode=0
Sep 17 11:18:59.469382 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.469258    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"
Sep 17 11:18:59.469382 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.469291    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd" event={"ID":"59b421d6-b8ce-4b1a-b6fe-98d20789ae46","Type":"ContainerDied","Data":"cff834d1ab87c98a7e54aeaaba155f00000d57692b3aa8277a34c8619d987ff6"}
Sep 17 11:18:59.469382 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.469331    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd" event={"ID":"59b421d6-b8ce-4b1a-b6fe-98d20789ae46","Type":"ContainerDied","Data":"c273a63144f6e7db00c871f5badb499fba5a864a36b4ab489849e27c00506f2e"}
Sep 17 11:18:59.469382 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.469355    2562 scope.go:117] "RemoveContainer" containerID="cff834d1ab87c98a7e54aeaaba155f00000d57692b3aa8277a34c8619d987ff6"
Sep 17 11:18:59.492150 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.491522    2562 scope.go:117] "RemoveContainer" containerID="cff834d1ab87c98a7e54aeaaba155f00000d57692b3aa8277a34c8619d987ff6"
Sep 17 11:18:59.492150 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:18:59.492099    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff834d1ab87c98a7e54aeaaba155f00000d57692b3aa8277a34c8619d987ff6\": container with ID starting with cff834d1ab87c98a7e54aeaaba155f00000d57692b3aa8277a34c8619d987ff6 not found: ID does not exist" containerID="cff834d1ab87c98a7e54aeaaba155f00000d57692b3aa8277a34c8619d987ff6"
Sep 17 11:18:59.492150 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.492138    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff834d1ab87c98a7e54aeaaba155f00000d57692b3aa8277a34c8619d987ff6"} err="failed to get container status \"cff834d1ab87c98a7e54aeaaba155f00000d57692b3aa8277a34c8619d987ff6\": rpc error: code = NotFound desc = could not find container \"cff834d1ab87c98a7e54aeaaba155f00000d57692b3aa8277a34c8619d987ff6\": container with ID starting with cff834d1ab87c98a7e54aeaaba155f00000d57692b3aa8277a34c8619d987ff6 not found: ID does not exist"
Sep 17 11:18:59.502834 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.502786    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"]
Sep 17 11:18:59.510002 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.509971    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8287f/sub-test-model-b8287f-kserve-68d4fbbcdf-g4ckd"]
Sep 17 11:18:59.562489 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.562452    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:59.562489 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.562485    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:59.562489 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.562496    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:59.562714 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.562507    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:59.562714 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.562518    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6fnv\" (UniqueName: \"kubernetes.io/projected/59b421d6-b8ce-4b1a-b6fe-98d20789ae46-kube-api-access-t6fnv\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:18:59.754829 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:18:59.754737    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:00.010780 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:00.010656    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:00.456004 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:00.455969    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a67cda4-e481-493e-9276-466ab4e2aa45" path="/var/lib/kubelet/pods/2a67cda4-e481-493e-9276-466ab4e2aa45/volumes"
Sep 17 11:19:00.456400 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:00.456387    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59b421d6-b8ce-4b1a-b6fe-98d20789ae46" path="/var/lib/kubelet/pods/59b421d6-b8ce-4b1a-b6fe-98d20789ae46/volumes"
Sep 17 11:19:03.938142 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:03.938093    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:19:03.938600 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:03.938185    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:19:03.950656 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:03.950627    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:19:04.510404 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:04.510368    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:19:05.848655 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:05.848619    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:19:11.547622 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.547589    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"]
Sep 17 11:19:11.548049 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.548035    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a67cda4-e481-493e-9276-466ab4e2aa45" containerName="main"
Sep 17 11:19:11.548097 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.548051    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a67cda4-e481-493e-9276-466ab4e2aa45" containerName="main"
Sep 17 11:19:11.548097 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.548064    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b421d6-b8ce-4b1a-b6fe-98d20789ae46" containerName="main"
Sep 17 11:19:11.548097 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.548069    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b421d6-b8ce-4b1a-b6fe-98d20789ae46" containerName="main"
Sep 17 11:19:11.548195 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.548135    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b421d6-b8ce-4b1a-b6fe-98d20789ae46" containerName="main"
Sep 17 11:19:11.548195 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.548144    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a67cda4-e481-493e-9276-466ab4e2aa45" containerName="main"
Sep 17 11:19:11.552445 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.552428    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.557277 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.557246    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8287f\"/\"default-dockercfg-jpfb4\""
Sep 17 11:19:11.557414 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.557316    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8287f\"/\"rate-test-model-b8287f-kserve-self-signed-certs\""
Sep 17 11:19:11.557414 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.557347    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8287f\"/\"openshift-service-ca.crt\""
Sep 17 11:19:11.557414 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.557317    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-b8287f\"/\"kube-root-ca.crt\""
Sep 17 11:19:11.577353 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.577325    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm9lx\" (UniqueName: \"kubernetes.io/projected/a2d8bb67-1d19-46bd-8447-23d3a7527d74-kube-api-access-sm9lx\") pod \"rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.577483 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.577377    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-dshm\") pod \"rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.577483 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.577428    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-tmp-dir\") pod \"rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.577483 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.577472    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d8bb67-1d19-46bd-8447-23d3a7527d74-tls-certs\") pod \"rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.577643 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.577503    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-home\") pod \"rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.577643 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.577526    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-model-cache\") pod \"rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.582532 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.582509    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"]
Sep 17 11:19:11.678553 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.678511    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm9lx\" (UniqueName: \"kubernetes.io/projected/a2d8bb67-1d19-46bd-8447-23d3a7527d74-kube-api-access-sm9lx\") pod \"rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.678738 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.678567    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-dshm\") pod \"rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.678738 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.678588    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-tmp-dir\") pod \"rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.678738 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.678612    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d8bb67-1d19-46bd-8447-23d3a7527d74-tls-certs\") pod \"rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.678738 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.678632    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-home\") pod \"rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.678738 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.678682    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-model-cache\") pod \"rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.679140 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.679120    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-home\") pod \"rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.679222 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.679159    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-tmp-dir\") pod \"rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.679222 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.679176    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-model-cache\") pod \"rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.680914 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.680894    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-dshm\") pod \"rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.681066 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.681051    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d8bb67-1d19-46bd-8447-23d3a7527d74-tls-certs\") pod \"rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.688046 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.688022    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm9lx\" (UniqueName: \"kubernetes.io/projected/a2d8bb67-1d19-46bd-8447-23d3a7527d74-kube-api-access-sm9lx\") pod \"rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") " pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:11.739779 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.739750    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:11.862468 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:11.862378    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:12.155601 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:12.155496    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"]
Sep 17 11:19:12.158029 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:19:12.158004    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2d8bb67_1d19_46bd_8447_23d3a7527d74.slice/crio-e85c6649e685389899ab82b81d7e53807d278058c923938b2659209ada328df8 WatchSource:0}: Error finding container e85c6649e685389899ab82b81d7e53807d278058c923938b2659209ada328df8: Status 404 returned error can't find the container with id e85c6649e685389899ab82b81d7e53807d278058c923938b2659209ada328df8
Sep 17 11:19:12.545182 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:12.545149    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg" event={"ID":"a2d8bb67-1d19-46bd-8447-23d3a7527d74","Type":"ContainerStarted","Data":"24d1c024df5ac9fcf092cef6775207104d2ea7a30ff2833211420b9cd510d556"}
Sep 17 11:19:12.545377 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:12.545192    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg" event={"ID":"a2d8bb67-1d19-46bd-8447-23d3a7527d74","Type":"ContainerStarted","Data":"e85c6649e685389899ab82b81d7e53807d278058c923938b2659209ada328df8"}
Sep 17 11:19:12.568903 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:12.568858    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg" podStartSLOduration=1.568842012 podStartE2EDuration="1.568842012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:19:12.567969705 +0000 UTC m=+1546.666015271" watchObservedRunningTime="2026-09-17 11:19:12.568842012 +0000 UTC m=+1546.666887569"
Sep 17 11:19:16.323482 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:16.323450    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:19:20.933862 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:20.933807    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:21.863099 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:21.863063    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:21.863320 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:21.863133    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:21.875396 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:21.875371    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:22.608632 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:22.608602    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:19:26.879001 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:26.878920    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:26.952163 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:26.952128    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:27.118258 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:27.118220    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:27.651125 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:27.651084    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:28.115061 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:28.115028    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:28.473046 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:28.472963    2562 pod_container_manager_linux.go:217] "Failed to delete cgroup paths" cgroupName=["kubepods","besteffort","pod2a67cda4-e481-493e-9276-466ab4e2aa45"] err="unable to destroy cgroup paths for cgroup [kubepods besteffort pod2a67cda4-e481-493e-9276-466ab4e2aa45] : Timed out while waiting for systemd to remove kubepods-besteffort-pod2a67cda4_e481_493e_9276_466ab4e2aa45.slice"
Sep 17 11:19:29.946935 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:29.946901    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:30.471581 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:30.471548    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:30.570372 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:30.570337    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:30.889204 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:30.889151    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:31.954054 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:31.954016    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:32.736440 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:32.736409    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:33.513060 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:33.513023    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"]
Sep 17 11:19:33.516843 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:33.516811    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:19:33.519827 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:33.519791    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c6abf3-serving-cert\""
Sep 17 11:19:33.530602 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:33.530574    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"]
Sep 17 11:19:33.551441 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:33.551417    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:33.591287 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:33.591258    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-openshift-service-ca\") pod \"maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r\" (UID: \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:19:33.591438 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:33.591321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-maas-api-tls\") pod \"maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r\" (UID: \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:19:33.591491 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:33.591457    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-metrics-certs\") pod \"maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r\" (UID: \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:19:33.591532 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:33.591501    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v48qz\" (UniqueName: \"kubernetes.io/projected/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-kube-api-access-v48qz\") pod \"maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r\" (UID: \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:19:33.650283 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:33.650250    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:33.692642 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:33.692615    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-maas-api-tls\") pod \"maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r\" (UID: \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:19:33.692811 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:33.692680    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-metrics-certs\") pod \"maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r\" (UID: \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:19:33.692811 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:33.692706    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v48qz\" (UniqueName: \"kubernetes.io/projected/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-kube-api-access-v48qz\") pod \"maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r\" (UID: \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:19:33.692959 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:19:33.692804    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-serving-cert: secret "maas-api-e2e-api-a-c6abf3-serving-cert" not found
Sep 17 11:19:33.692959 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:33.692871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-openshift-service-ca\") pod \"maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r\" (UID: \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:19:33.692959 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:19:33.692949    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-maas-api-tls podName:0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd nodeName:}" failed. No retries permitted until 2026-09-17 11:19:34.192927334 +0000 UTC m=+1568.290972898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-maas-api-tls") pod "maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r" (UID: "0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd") : secret "maas-api-e2e-api-a-c6abf3-serving-cert" not found
Sep 17 11:19:33.693539 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:33.693518    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-openshift-service-ca\") pod \"maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r\" (UID: \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:19:33.695171 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:33.695141    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-metrics-certs\") pod \"maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r\" (UID: \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:19:33.704562 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:33.704537    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v48qz\" (UniqueName: \"kubernetes.io/projected/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-kube-api-access-v48qz\") pod \"maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r\" (UID: \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:19:33.774191 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:33.774111    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:34.064989 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:34.064958    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:34.198441 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:34.198408    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-maas-api-tls\") pod \"maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r\" (UID: \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:19:34.200779 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:34.200751    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-maas-api-tls\") pod \"maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r\" (UID: \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:19:34.391520 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:34.391441    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:34.427179 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:34.427143    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:19:34.703430 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:34.703403    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"]
Sep 17 11:19:34.704850 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:19:34.704811    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dfa7eb3_fe3b_4a1f_a98d_4dca928d72bd.slice/crio-36297ceed6d8ea02ee68ec6108e165d61bdaacf418d7be83ccd0a713ad039d61 WatchSource:0}: Error finding container 36297ceed6d8ea02ee68ec6108e165d61bdaacf418d7be83ccd0a713ad039d61: Status 404 returned error can't find the container with id 36297ceed6d8ea02ee68ec6108e165d61bdaacf418d7be83ccd0a713ad039d61
Sep 17 11:19:35.666398 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:35.666358    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r" event={"ID":"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd","Type":"ContainerStarted","Data":"0b4efb6bc821b996d5228956fc400a49dad0cbf4a29fc5f80245cea144170afb"}
Sep 17 11:19:35.666608 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:35.666451    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r" event={"ID":"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd","Type":"ContainerStarted","Data":"36297ceed6d8ea02ee68ec6108e165d61bdaacf418d7be83ccd0a713ad039d61"}
Sep 17 11:19:35.666608 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:35.666490    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:19:35.689101 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:35.689049    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r" podStartSLOduration=2.129723098 podStartE2EDuration="2.689034632s" totalImagesPullingTime="559.311534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:19:35.686618405 +0000 UTC m=+1569.784663971" watchObservedRunningTime="2026-09-17 11:19:35.689034632 +0000 UTC m=+1569.787080197"
Sep 17 11:19:39.414495 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:39.414457    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:19:41.678049 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:41.678016    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:19:56.088844 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:56.088797    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:56.722410 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:56.722377    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:19:57.137022 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:57.136991    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:57.948615 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:57.948580    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:58.779660 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:58.779630    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:19:59.482744 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:19:59.482702    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:00.468586 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:00.468547    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:00.946862 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:00.946829    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:01.304240 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:01.304206    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:01.646562 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:01.646481    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:03.471928 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:03.471895    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:04.187365 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:04.187334    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:04.286494 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:04.286463    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:05.257783 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:05.257755    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:06.008335 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:06.008303    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:06.825434 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:06.825396    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:07.526786 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:07.526752    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:20:07.870217 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:07.870138    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"]
Sep 17 11:20:07.873904 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:07.873888    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"
Sep 17 11:20:07.876587 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:07.876565    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9f97d7-serving-cert\""
Sep 17 11:20:07.877162 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:07.877139    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:07.887416 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:07.887388    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"]
Sep 17 11:20:07.931990 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:07.931956    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-openshift-service-ca\") pod \"maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9\" (UID: \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"
Sep 17 11:20:07.932157 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:07.932021    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-metrics-certs\") pod \"maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9\" (UID: \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"
Sep 17 11:20:07.932157 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:07.932085    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw5df\" (UniqueName: \"kubernetes.io/projected/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-kube-api-access-fw5df\") pod \"maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9\" (UID: \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"
Sep 17 11:20:07.932157 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:07.932129    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-maas-api-tls\") pod \"maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9\" (UID: \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"
Sep 17 11:20:07.964917 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:07.964886    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:08.032930 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:08.032897    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-openshift-service-ca\") pod \"maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9\" (UID: \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"
Sep 17 11:20:08.033112 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:08.032949    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-metrics-certs\") pod \"maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9\" (UID: \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"
Sep 17 11:20:08.033112 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:08.032981    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw5df\" (UniqueName: \"kubernetes.io/projected/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-kube-api-access-fw5df\") pod \"maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9\" (UID: \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"
Sep 17 11:20:08.033112 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:08.033003    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-maas-api-tls\") pod \"maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9\" (UID: \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"
Sep 17 11:20:08.033560 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:08.033539    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-openshift-service-ca\") pod \"maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9\" (UID: \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"
Sep 17 11:20:08.035586 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:08.035560    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-maas-api-tls\") pod \"maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9\" (UID: \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"
Sep 17 11:20:08.035586 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:08.035577    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-metrics-certs\") pod \"maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9\" (UID: \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"
Sep 17 11:20:08.040957 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:08.040936    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw5df\" (UniqueName: \"kubernetes.io/projected/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-kube-api-access-fw5df\") pod \"maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9\" (UID: \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"
Sep 17 11:20:08.065597 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:08.065557    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:08.184884 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:08.184792    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"
Sep 17 11:20:08.392716 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:08.392681    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:08.478374 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:08.478354    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"]
Sep 17 11:20:08.481283 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:20:08.481258    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169c2f46_9f99_48c4_b90c_1839dc5c2fc1.slice/crio-d628f0ebe12becd25a7fcbaa36123e5e52e2fab7900e1541369963a131c075ec WatchSource:0}: Error finding container d628f0ebe12becd25a7fcbaa36123e5e52e2fab7900e1541369963a131c075ec: Status 404 returned error can't find the container with id d628f0ebe12becd25a7fcbaa36123e5e52e2fab7900e1541369963a131c075ec
Sep 17 11:20:08.499943 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:08.499921    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:08.841706 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:08.841671    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9" event={"ID":"169c2f46-9f99-48c4-b90c-1839dc5c2fc1","Type":"ContainerStarted","Data":"d628f0ebe12becd25a7fcbaa36123e5e52e2fab7900e1541369963a131c075ec"}
Sep 17 11:20:08.942039 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:08.942011    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:09.735115 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:09.735051    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:09.848963 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:09.848927    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9" event={"ID":"169c2f46-9f99-48c4-b90c-1839dc5c2fc1","Type":"ContainerStarted","Data":"3e79b6c4e83fb4872e0ba06da28ebf4f94021f17ecca92127adb6e9890979df2"}
Sep 17 11:20:09.849128 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:09.849013    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"
Sep 17 11:20:09.890735 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:09.890669    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9" podStartSLOduration=2.438803065 podStartE2EDuration="2.890653807s" totalImagesPullingTime="451.850742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:20:09.888994802 +0000 UTC m=+1603.987040372" watchObservedRunningTime="2026-09-17 11:20:09.890653807 +0000 UTC m=+1603.988699364"
Sep 17 11:20:10.759043 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:10.759009    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:14.211141 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:14.211109    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"]
Sep 17 11:20:14.231547 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:14.231515    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:15.107605 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:15.107575    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"]
Sep 17 11:20:15.174919 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:15.174889    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:15.551834 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:15.551784    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:15.862138 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:15.862070    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"
Sep 17 11:20:15.865364 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:15.865347    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:15.985268 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:15.985228    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:18.463921 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:18.463891    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:19.229225 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:19.229150    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:19.334560 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:19.334524    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:20.316267 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:20.316229    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:21.333783 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:21.333745    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:22.540283 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:22.540251    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:23.364214 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:23.364178    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:24.014609 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.014581    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:24.043577 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.043544    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"]
Sep 17 11:20:24.047472 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.047450    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:20:24.051527 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.051502    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-31fca0-serving-cert\""
Sep 17 11:20:24.063648 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.063616    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"]
Sep 17 11:20:24.096397 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.096360    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kwxj\" (UniqueName: \"kubernetes.io/projected/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-kube-api-access-6kwxj\") pod \"maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs\" (UID: \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:20:24.096558 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.096441    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs\" (UID: \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:20:24.096558 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.096463    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-metrics-certs\") pod \"maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs\" (UID: \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:20:24.096558 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.096506    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs\" (UID: \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:20:24.117000 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.116969    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:24.197434 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.197399    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs\" (UID: \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:20:24.197434 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.197434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-metrics-certs\") pod \"maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs\" (UID: \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:20:24.197650 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.197490    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs\" (UID: \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:20:24.197650 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.197555    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kwxj\" (UniqueName: \"kubernetes.io/projected/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-kube-api-access-6kwxj\") pod \"maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs\" (UID: \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:20:24.197718 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:20:24.197650    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-serving-cert: secret "maas-api-e2e-ipp-a-31fca0-serving-cert" not found
Sep 17 11:20:24.197753 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:20:24.197727    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-maas-api-tls podName:d5c3fc04-6aba-4eaf-9016-f97c6f627fcd nodeName:}" failed. No retries permitted until 2026-09-17 11:20:24.697708256 +0000 UTC m=+1618.795753801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-maas-api-tls") pod "maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs" (UID: "d5c3fc04-6aba-4eaf-9016-f97c6f627fcd") : secret "maas-api-e2e-ipp-a-31fca0-serving-cert" not found
Sep 17 11:20:24.198222 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.198200    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs\" (UID: \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:20:24.200136 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.200111    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-metrics-certs\") pod \"maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs\" (UID: \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:20:24.207779 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.207751    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kwxj\" (UniqueName: \"kubernetes.io/projected/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-kube-api-access-6kwxj\") pod \"maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs\" (UID: \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:20:24.214586 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.214557    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:24.701744 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.701695    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs\" (UID: \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:20:24.701963 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:20:24.701901    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-serving-cert: secret "maas-api-e2e-ipp-a-31fca0-serving-cert" not found
Sep 17 11:20:24.702014 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:20:24.701977    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-maas-api-tls podName:d5c3fc04-6aba-4eaf-9016-f97c6f627fcd nodeName:}" failed. No retries permitted until 2026-09-17 11:20:25.701959696 +0000 UTC m=+1619.800005246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-maas-api-tls") pod "maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs" (UID: "d5c3fc04-6aba-4eaf-9016-f97c6f627fcd") : secret "maas-api-e2e-ipp-a-31fca0-serving-cert" not found
Sep 17 11:20:24.837426 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.837393    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:24.926138 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:24.926103    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:25.361800 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:25.361765    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:25.711323 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:25.711217    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs\" (UID: \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:20:25.713527 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:25.713505    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs\" (UID: \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:20:25.860049 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:25.860009    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:20:26.137763 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:26.137732    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"]
Sep 17 11:20:26.138943 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:20:26.138919    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c3fc04_6aba_4eaf_9016_f97c6f627fcd.slice/crio-e4fa9d1f3eb078ba39f609051a3a2adc2d28e5b47bea463c014977ee7900f7b4 WatchSource:0}: Error finding container e4fa9d1f3eb078ba39f609051a3a2adc2d28e5b47bea463c014977ee7900f7b4: Status 404 returned error can't find the container with id e4fa9d1f3eb078ba39f609051a3a2adc2d28e5b47bea463c014977ee7900f7b4
Sep 17 11:20:26.217905 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:26.217871    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:26.738675 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:26.738641    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:20:26.940674 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:26.940630    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs" event={"ID":"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd","Type":"ContainerStarted","Data":"7011dfe270c7e94fdedadd9b7a112d627e5b81ee991c03f95df8f5820696c2f5"}
Sep 17 11:20:26.940947 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:26.940684    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs" event={"ID":"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd","Type":"ContainerStarted","Data":"e4fa9d1f3eb078ba39f609051a3a2adc2d28e5b47bea463c014977ee7900f7b4"}
Sep 17 11:20:26.940947 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:26.940758    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:20:26.963123 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:26.963069    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs" podStartSLOduration=2.61080922 podStartE2EDuration="2.963053704s" totalImagesPullingTime="352.244484ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:20:26.960384542 +0000 UTC m=+1621.058430108" watchObservedRunningTime="2026-09-17 11:20:26.963053704 +0000 UTC m=+1621.061099270"
Sep 17 11:20:27.462391 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:27.462356    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:28.462982 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:28.462950    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:29.229879 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.229811    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7" podUID="f28fc50b-e566-4644-b648-bbd99a1436ef" containerName="main" containerID="cri-o://7194443c246f6de9cefc306e56df0c6dd8b661b5d1a507acb418ba458138d519" gracePeriod=45
Sep 17 11:20:29.231499 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.231474    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:29.488314 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.488257    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:20:29.550353 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.550317    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-tmp-dir\") pod \"f28fc50b-e566-4644-b648-bbd99a1436ef\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") "
Sep 17 11:20:29.550557 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.550414    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwwhs\" (UniqueName: \"kubernetes.io/projected/f28fc50b-e566-4644-b648-bbd99a1436ef-kube-api-access-gwwhs\") pod \"f28fc50b-e566-4644-b648-bbd99a1436ef\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") "
Sep 17 11:20:29.550557 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.550442    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-model-cache\") pod \"f28fc50b-e566-4644-b648-bbd99a1436ef\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") "
Sep 17 11:20:29.550557 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.550483    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f28fc50b-e566-4644-b648-bbd99a1436ef-tls-certs\") pod \"f28fc50b-e566-4644-b648-bbd99a1436ef\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") "
Sep 17 11:20:29.550557 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.550531    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-dshm\") pod \"f28fc50b-e566-4644-b648-bbd99a1436ef\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") "
Sep 17 11:20:29.550557 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.550545    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-home\") pod \"f28fc50b-e566-4644-b648-bbd99a1436ef\" (UID: \"f28fc50b-e566-4644-b648-bbd99a1436ef\") "
Sep 17 11:20:29.550867 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.550626    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f28fc50b-e566-4644-b648-bbd99a1436ef" (UID: "f28fc50b-e566-4644-b648-bbd99a1436ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:20:29.550867 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.550700    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "f28fc50b-e566-4644-b648-bbd99a1436ef" (UID: "f28fc50b-e566-4644-b648-bbd99a1436ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:20:29.550998 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.550919    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-home" (OuterVolumeSpecName: "home") pod "f28fc50b-e566-4644-b648-bbd99a1436ef" (UID: "f28fc50b-e566-4644-b648-bbd99a1436ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:20:29.550998 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.550922    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:20:29.550998 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.550959    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:20:29.552782 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.552748    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f28fc50b-e566-4644-b648-bbd99a1436ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f28fc50b-e566-4644-b648-bbd99a1436ef" (UID: "f28fc50b-e566-4644-b648-bbd99a1436ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:20:29.553032 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.553009    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-dshm" (OuterVolumeSpecName: "dshm") pod "f28fc50b-e566-4644-b648-bbd99a1436ef" (UID: "f28fc50b-e566-4644-b648-bbd99a1436ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:20:29.553136 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.553045    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28fc50b-e566-4644-b648-bbd99a1436ef-kube-api-access-gwwhs" (OuterVolumeSpecName: "kube-api-access-gwwhs") pod "f28fc50b-e566-4644-b648-bbd99a1436ef" (UID: "f28fc50b-e566-4644-b648-bbd99a1436ef"). InnerVolumeSpecName "kube-api-access-gwwhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:20:29.651804 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.651763    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwwhs\" (UniqueName: \"kubernetes.io/projected/f28fc50b-e566-4644-b648-bbd99a1436ef-kube-api-access-gwwhs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:20:29.651804 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.651799    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f28fc50b-e566-4644-b648-bbd99a1436ef-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:20:29.651804 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.651811    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:20:29.652085 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.651844    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f28fc50b-e566-4644-b648-bbd99a1436ef-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:20:29.956586 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.956554    2562 generic.go:356] "Generic (PLEG): container finished" podID="f28fc50b-e566-4644-b648-bbd99a1436ef" containerID="7194443c246f6de9cefc306e56df0c6dd8b661b5d1a507acb418ba458138d519" exitCode=0
Sep 17 11:20:29.956768 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.956600    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"
Sep 17 11:20:29.956768 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.956641    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7" event={"ID":"f28fc50b-e566-4644-b648-bbd99a1436ef","Type":"ContainerDied","Data":"7194443c246f6de9cefc306e56df0c6dd8b661b5d1a507acb418ba458138d519"}
Sep 17 11:20:29.956768 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.956684    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7" event={"ID":"f28fc50b-e566-4644-b648-bbd99a1436ef","Type":"ContainerDied","Data":"4042943d734a72ebf1a5d04b97443a1836ff54baa0192407cafc6923f4591807"}
Sep 17 11:20:29.956768 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.956702    2562 scope.go:117] "RemoveContainer" containerID="7194443c246f6de9cefc306e56df0c6dd8b661b5d1a507acb418ba458138d519"
Sep 17 11:20:29.966241 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.966218    2562 scope.go:117] "RemoveContainer" containerID="7194443c246f6de9cefc306e56df0c6dd8b661b5d1a507acb418ba458138d519"
Sep 17 11:20:29.966524 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:20:29.966505    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7194443c246f6de9cefc306e56df0c6dd8b661b5d1a507acb418ba458138d519\": container with ID starting with 7194443c246f6de9cefc306e56df0c6dd8b661b5d1a507acb418ba458138d519 not found: ID does not exist" containerID="7194443c246f6de9cefc306e56df0c6dd8b661b5d1a507acb418ba458138d519"
Sep 17 11:20:29.966588 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.966531    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7194443c246f6de9cefc306e56df0c6dd8b661b5d1a507acb418ba458138d519"} err="failed to get container status \"7194443c246f6de9cefc306e56df0c6dd8b661b5d1a507acb418ba458138d519\": rpc error: code = NotFound desc = could not find container \"7194443c246f6de9cefc306e56df0c6dd8b661b5d1a507acb418ba458138d519\": container with ID starting with 7194443c246f6de9cefc306e56df0c6dd8b661b5d1a507acb418ba458138d519 not found: ID does not exist"
Sep 17 11:20:29.981202 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.981170    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"]
Sep 17 11:20:29.984103 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:29.984078    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-026f86/rate-test-model-026f86-kserve-f99dddb79-2spr7"]
Sep 17 11:20:30.032883 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.032844    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:30.126952 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.126908    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg" podUID="a2d8bb67-1d19-46bd-8447-23d3a7527d74" containerName="main" containerID="cri-o://24d1c024df5ac9fcf092cef6775207104d2ea7a30ff2833211420b9cd510d556" gracePeriod=45
Sep 17 11:20:30.388669 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.388643    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:20:30.456148 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.456115    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f28fc50b-e566-4644-b648-bbd99a1436ef" path="/var/lib/kubelet/pods/f28fc50b-e566-4644-b648-bbd99a1436ef/volumes"
Sep 17 11:20:30.459083 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.459060    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-tmp-dir\") pod \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") "
Sep 17 11:20:30.459223 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.459156    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sm9lx\" (UniqueName: \"kubernetes.io/projected/a2d8bb67-1d19-46bd-8447-23d3a7527d74-kube-api-access-sm9lx\") pod \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") "
Sep 17 11:20:30.459223 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.459185    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-home\") pod \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") "
Sep 17 11:20:30.459340 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.459232    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d8bb67-1d19-46bd-8447-23d3a7527d74-tls-certs\") pod \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") "
Sep 17 11:20:30.459340 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.459290    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-dshm\") pod \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") "
Sep 17 11:20:30.459340 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.459319    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-model-cache\") pod \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\" (UID: \"a2d8bb67-1d19-46bd-8447-23d3a7527d74\") "
Sep 17 11:20:30.459640 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.459311    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2d8bb67-1d19-46bd-8447-23d3a7527d74" (UID: "a2d8bb67-1d19-46bd-8447-23d3a7527d74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:20:30.459761 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.459614    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2d8bb67-1d19-46bd-8447-23d3a7527d74" (UID: "a2d8bb67-1d19-46bd-8447-23d3a7527d74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:20:30.459761 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.459671    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-home" (OuterVolumeSpecName: "home") pod "a2d8bb67-1d19-46bd-8447-23d3a7527d74" (UID: "a2d8bb67-1d19-46bd-8447-23d3a7527d74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:20:30.461418 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.461391    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d8bb67-1d19-46bd-8447-23d3a7527d74-kube-api-access-sm9lx" (OuterVolumeSpecName: "kube-api-access-sm9lx") pod "a2d8bb67-1d19-46bd-8447-23d3a7527d74" (UID: "a2d8bb67-1d19-46bd-8447-23d3a7527d74"). InnerVolumeSpecName "kube-api-access-sm9lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:20:30.461780 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.461761    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-dshm" (OuterVolumeSpecName: "dshm") pod "a2d8bb67-1d19-46bd-8447-23d3a7527d74" (UID: "a2d8bb67-1d19-46bd-8447-23d3a7527d74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:20:30.461871 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.461784    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2d8bb67-1d19-46bd-8447-23d3a7527d74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2d8bb67-1d19-46bd-8447-23d3a7527d74" (UID: "a2d8bb67-1d19-46bd-8447-23d3a7527d74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:20:30.561144 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.561108    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d8bb67-1d19-46bd-8447-23d3a7527d74-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:20:30.561144 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.561142    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:20:30.561549 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.561154    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:20:30.561549 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.561165    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:20:30.561549 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.561176    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm9lx\" (UniqueName: \"kubernetes.io/projected/a2d8bb67-1d19-46bd-8447-23d3a7527d74-kube-api-access-sm9lx\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:20:30.561549 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.561190    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d8bb67-1d19-46bd-8447-23d3a7527d74-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:20:30.924976 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.924894    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:30.963014 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.962976    2562 generic.go:356] "Generic (PLEG): container finished" podID="a2d8bb67-1d19-46bd-8447-23d3a7527d74" containerID="24d1c024df5ac9fcf092cef6775207104d2ea7a30ff2833211420b9cd510d556" exitCode=0
Sep 17 11:20:30.963186 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.963035    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"
Sep 17 11:20:30.963186 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.963058    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg" event={"ID":"a2d8bb67-1d19-46bd-8447-23d3a7527d74","Type":"ContainerDied","Data":"24d1c024df5ac9fcf092cef6775207104d2ea7a30ff2833211420b9cd510d556"}
Sep 17 11:20:30.963186 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.963099    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg" event={"ID":"a2d8bb67-1d19-46bd-8447-23d3a7527d74","Type":"ContainerDied","Data":"e85c6649e685389899ab82b81d7e53807d278058c923938b2659209ada328df8"}
Sep 17 11:20:30.963186 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.963117    2562 scope.go:117] "RemoveContainer" containerID="24d1c024df5ac9fcf092cef6775207104d2ea7a30ff2833211420b9cd510d556"
Sep 17 11:20:30.972999 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.972980    2562 scope.go:117] "RemoveContainer" containerID="24d1c024df5ac9fcf092cef6775207104d2ea7a30ff2833211420b9cd510d556"
Sep 17 11:20:30.973273 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:20:30.973252    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d1c024df5ac9fcf092cef6775207104d2ea7a30ff2833211420b9cd510d556\": container with ID starting with 24d1c024df5ac9fcf092cef6775207104d2ea7a30ff2833211420b9cd510d556 not found: ID does not exist" containerID="24d1c024df5ac9fcf092cef6775207104d2ea7a30ff2833211420b9cd510d556"
Sep 17 11:20:30.973319 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.973281    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d1c024df5ac9fcf092cef6775207104d2ea7a30ff2833211420b9cd510d556"} err="failed to get container status \"24d1c024df5ac9fcf092cef6775207104d2ea7a30ff2833211420b9cd510d556\": rpc error: code = NotFound desc = could not find container \"24d1c024df5ac9fcf092cef6775207104d2ea7a30ff2833211420b9cd510d556\": container with ID starting with 24d1c024df5ac9fcf092cef6775207104d2ea7a30ff2833211420b9cd510d556 not found: ID does not exist"
Sep 17 11:20:30.986274 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.986247    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"]
Sep 17 11:20:30.989995 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:30.989970    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b8287f/rate-test-model-b8287f-kserve-8467b5bbcf-kt4cg"]
Sep 17 11:20:31.655633 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:31.655593    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:32.456465 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:32.456423    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2d8bb67-1d19-46bd-8447-23d3a7527d74" path="/var/lib/kubelet/pods/a2d8bb67-1d19-46bd-8447-23d3a7527d74/volumes"
Sep 17 11:20:32.814089 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:32.814052    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:32.951742 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:32.951713    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:20:33.833057 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:33.833024    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:34.650372 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:34.650342    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:37.154390 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:37.154355    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:42.786215 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:42.786185    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"]
Sep 17 11:20:42.786591 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:42.786389    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r" podUID="0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd" containerName="maas-api" containerID="cri-o://0b4efb6bc821b996d5228956fc400a49dad0cbf4a29fc5f80245cea144170afb" gracePeriod=30
Sep 17 11:20:42.817248 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:42.817215    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:42.916652 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:42.916615    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:43.020091 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.020065    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:43.030983 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.030954    2562 generic.go:356] "Generic (PLEG): container finished" podID="0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd" containerID="0b4efb6bc821b996d5228956fc400a49dad0cbf4a29fc5f80245cea144170afb" exitCode=0
Sep 17 11:20:43.031107 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.030982    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r" event={"ID":"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd","Type":"ContainerDied","Data":"0b4efb6bc821b996d5228956fc400a49dad0cbf4a29fc5f80245cea144170afb"}
Sep 17 11:20:43.031107 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.031037    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r" event={"ID":"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd","Type":"ContainerDied","Data":"36297ceed6d8ea02ee68ec6108e165d61bdaacf418d7be83ccd0a713ad039d61"}
Sep 17 11:20:43.031107 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.031058    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36297ceed6d8ea02ee68ec6108e165d61bdaacf418d7be83ccd0a713ad039d61"
Sep 17 11:20:43.041404 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.041348    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:20:43.180414 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.180373    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:43.184451 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.184428    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-maas-api-tls\") pod \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\" (UID: \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\") "
Sep 17 11:20:43.184571 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.184488    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-openshift-service-ca\") pod \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\" (UID: \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\") "
Sep 17 11:20:43.184640 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.184570    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v48qz\" (UniqueName: \"kubernetes.io/projected/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-kube-api-access-v48qz\") pod \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\" (UID: \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\") "
Sep 17 11:20:43.184640 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.184601    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-metrics-certs\") pod \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\" (UID: \"0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd\") "
Sep 17 11:20:43.184877 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.184851    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd" (UID: "0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:20:43.186697 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.186666    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd" (UID: "0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:20:43.186792 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.186734    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd" (UID: "0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:20:43.186792 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.186741    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-kube-api-access-v48qz" (OuterVolumeSpecName: "kube-api-access-v48qz") pod "0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd" (UID: "0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd"). InnerVolumeSpecName "kube-api-access-v48qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:20:43.285552 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.285515    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:20:43.285552 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.285546    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v48qz\" (UniqueName: \"kubernetes.io/projected/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-kube-api-access-v48qz\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:20:43.285552 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.285557    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:20:43.285795 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.285565    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:20:43.583740 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.583706    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:43.630764 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.630723    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:20:43.682073 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:43.682043    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:44.036353 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:44.036321    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"
Sep 17 11:20:44.063776 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:44.063730    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"]
Sep 17 11:20:44.066349 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:44.066324    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c6abf3-6c5b859965-rsw4r"]
Sep 17 11:20:44.462994 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:44.462918    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd" path="/var/lib/kubelet/pods/0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd/volumes"
Sep 17 11:20:44.722341 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:44.722246    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:45.663309 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:45.663272    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:46.815611 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:46.815571    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:47.364364 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:47.364306    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:48.240110 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:48.240079    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:48.340565 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:48.340534    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:48.444022 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:48.443990    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:49.318762 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:49.318727    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:49.478732 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:49.478702    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:50.081204 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:50.081173    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:50.450149 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:50.450073    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:50.549240 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:50.549212    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:50.644498 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:50.644457    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:50.771178 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:50.771102    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:50.882449 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:50.882416    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:51.074088 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:51.074055    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:51.626496 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:51.626463    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:52.703630 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:52.703597    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:53.589338 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:53.589309    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:53.940373 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:53.940281    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:54.033020 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:54.032985    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:54.598295 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:54.598266    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:55.744039 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:55.744005    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:56.669053 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:56.669014    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:56.828588 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:56.828548    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:20:57.481832 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:57.481788    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:58.307102 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.307067    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"]
Sep 17 11:20:58.307780 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.307760    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f28fc50b-e566-4644-b648-bbd99a1436ef" containerName="main"
Sep 17 11:20:58.307970 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.307782    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f28fc50b-e566-4644-b648-bbd99a1436ef" containerName="main"
Sep 17 11:20:58.307970 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.307828    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2d8bb67-1d19-46bd-8447-23d3a7527d74" containerName="main"
Sep 17 11:20:58.307970 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.307838    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2d8bb67-1d19-46bd-8447-23d3a7527d74" containerName="main"
Sep 17 11:20:58.307970 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.307847    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd" containerName="maas-api"
Sep 17 11:20:58.307970 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.307856    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd" containerName="maas-api"
Sep 17 11:20:58.307970 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.307951    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f28fc50b-e566-4644-b648-bbd99a1436ef" containerName="main"
Sep 17 11:20:58.307970 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.307960    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2d8bb67-1d19-46bd-8447-23d3a7527d74" containerName="main"
Sep 17 11:20:58.307970 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.307974    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dfa7eb3-fe3b-4a1f-a98d-4dca928d72bd" containerName="maas-api"
Sep 17 11:20:58.311470 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.311448    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"
Sep 17 11:20:58.314243 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.314220    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-23a45b-serving-cert\""
Sep 17 11:20:58.323632 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.323605    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"]
Sep 17 11:20:58.360184 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.360116    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:58.433201 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.433161    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3e55b40-b624-4fef-b839-973748e993c6-metrics-certs\") pod \"maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9\" (UID: \"b3e55b40-b624-4fef-b839-973748e993c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"
Sep 17 11:20:58.433390 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.433267    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3e55b40-b624-4fef-b839-973748e993c6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9\" (UID: \"b3e55b40-b624-4fef-b839-973748e993c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"
Sep 17 11:20:58.433390 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.433293    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e55b40-b624-4fef-b839-973748e993c6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9\" (UID: \"b3e55b40-b624-4fef-b839-973748e993c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"
Sep 17 11:20:58.433390 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.433327    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xltsw\" (UniqueName: \"kubernetes.io/projected/b3e55b40-b624-4fef-b839-973748e993c6-kube-api-access-xltsw\") pod \"maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9\" (UID: \"b3e55b40-b624-4fef-b839-973748e993c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"
Sep 17 11:20:58.460942 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.460906    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:58.534140 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.534097    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3e55b40-b624-4fef-b839-973748e993c6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9\" (UID: \"b3e55b40-b624-4fef-b839-973748e993c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"
Sep 17 11:20:58.534348 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.534147    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e55b40-b624-4fef-b839-973748e993c6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9\" (UID: \"b3e55b40-b624-4fef-b839-973748e993c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"
Sep 17 11:20:58.534348 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.534198    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xltsw\" (UniqueName: \"kubernetes.io/projected/b3e55b40-b624-4fef-b839-973748e993c6-kube-api-access-xltsw\") pod \"maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9\" (UID: \"b3e55b40-b624-4fef-b839-973748e993c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"
Sep 17 11:20:58.534348 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.534230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3e55b40-b624-4fef-b839-973748e993c6-metrics-certs\") pod \"maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9\" (UID: \"b3e55b40-b624-4fef-b839-973748e993c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"
Sep 17 11:20:58.534986 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.534957    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e55b40-b624-4fef-b839-973748e993c6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9\" (UID: \"b3e55b40-b624-4fef-b839-973748e993c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"
Sep 17 11:20:58.536716 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.536693    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3e55b40-b624-4fef-b839-973748e993c6-metrics-certs\") pod \"maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9\" (UID: \"b3e55b40-b624-4fef-b839-973748e993c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"
Sep 17 11:20:58.536809 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.536721    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3e55b40-b624-4fef-b839-973748e993c6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9\" (UID: \"b3e55b40-b624-4fef-b839-973748e993c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"
Sep 17 11:20:58.543565 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.543544    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xltsw\" (UniqueName: \"kubernetes.io/projected/b3e55b40-b624-4fef-b839-973748e993c6-kube-api-access-xltsw\") pod \"maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9\" (UID: \"b3e55b40-b624-4fef-b839-973748e993c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"
Sep 17 11:20:58.560358 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.560231    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:58.622838 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.622788    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"
Sep 17 11:20:58.896605 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.896578    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"]
Sep 17 11:20:58.898825 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:20:58.898782    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e55b40_b624_4fef_b839_973748e993c6.slice/crio-55e9c1350dc0f06cb29961b04e48b6127efa5a00da224e9465bf891eab9729c9 WatchSource:0}: Error finding container 55e9c1350dc0f06cb29961b04e48b6127efa5a00da224e9465bf891eab9729c9: Status 404 returned error can't find the container with id 55e9c1350dc0f06cb29961b04e48b6127efa5a00da224e9465bf891eab9729c9
Sep 17 11:20:58.924232 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:58.924202    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:59.023962 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:59.023921    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:20:59.118149 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:59.118115    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9" event={"ID":"b3e55b40-b624-4fef-b839-973748e993c6","Type":"ContainerStarted","Data":"55e9c1350dc0f06cb29961b04e48b6127efa5a00da224e9465bf891eab9729c9"}
Sep 17 11:20:59.347808 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:20:59.347777    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:00.125560 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:00.125515    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9" event={"ID":"b3e55b40-b624-4fef-b839-973748e993c6","Type":"ContainerStarted","Data":"ee2b3ebb16d69087167cb0c7263475d5c323e720ff98d3d41d4b8bb44bb9fe6f"}
Sep 17 11:21:00.125765 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:00.125591    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"
Sep 17 11:21:00.145221 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:00.145164    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9" podStartSLOduration=1.687537721 podStartE2EDuration="2.145147725s" totalImagesPullingTime="457.610004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:21:00.142973946 +0000 UTC m=+1654.241019511" watchObservedRunningTime="2026-09-17 11:21:00.145147725 +0000 UTC m=+1654.243193291"
Sep 17 11:21:00.151080 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:00.151053    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:01.792405 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:01.792363    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:04.783721 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:04.783682    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:06.136390 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:06.136359    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"
Sep 17 11:21:07.514452 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.514420    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"]
Sep 17 11:21:07.514875 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.514617    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9" podUID="169c2f46-9f99-48c4-b90c-1839dc5c2fc1" containerName="maas-api" containerID="cri-o://3e79b6c4e83fb4872e0ba06da28ebf4f94021f17ecca92127adb6e9890979df2" gracePeriod=30
Sep 17 11:21:07.549514 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.549480    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:07.635802 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.635770    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:07.735966 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.735936    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:07.768586 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.768519    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"
Sep 17 11:21:07.813474 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.813442    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:07.824563 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.824539    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw5df\" (UniqueName: \"kubernetes.io/projected/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-kube-api-access-fw5df\") pod \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\" (UID: \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\") "
Sep 17 11:21:07.824710 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.824572    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-openshift-service-ca\") pod \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\" (UID: \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\") "
Sep 17 11:21:07.824710 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.824635    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-maas-api-tls\") pod \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\" (UID: \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\") "
Sep 17 11:21:07.824710 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.824654    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-metrics-certs\") pod \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\" (UID: \"169c2f46-9f99-48c4-b90c-1839dc5c2fc1\") "
Sep 17 11:21:07.825039 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.825006    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "169c2f46-9f99-48c4-b90c-1839dc5c2fc1" (UID: "169c2f46-9f99-48c4-b90c-1839dc5c2fc1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:21:07.826862 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.826826    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "169c2f46-9f99-48c4-b90c-1839dc5c2fc1" (UID: "169c2f46-9f99-48c4-b90c-1839dc5c2fc1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:21:07.826959 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.826925    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "169c2f46-9f99-48c4-b90c-1839dc5c2fc1" (UID: "169c2f46-9f99-48c4-b90c-1839dc5c2fc1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:21:07.826959 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.826939    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-kube-api-access-fw5df" (OuterVolumeSpecName: "kube-api-access-fw5df") pod "169c2f46-9f99-48c4-b90c-1839dc5c2fc1" (UID: "169c2f46-9f99-48c4-b90c-1839dc5c2fc1"). InnerVolumeSpecName "kube-api-access-fw5df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:21:07.925427 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.925396    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:21:07.925427 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.925422    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:21:07.925427 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.925434    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw5df\" (UniqueName: \"kubernetes.io/projected/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-kube-api-access-fw5df\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:21:07.925654 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:07.925449    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c2f46-9f99-48c4-b90c-1839dc5c2fc1-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:21:08.170327 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:08.170287    2562 generic.go:356] "Generic (PLEG): container finished" podID="169c2f46-9f99-48c4-b90c-1839dc5c2fc1" containerID="3e79b6c4e83fb4872e0ba06da28ebf4f94021f17ecca92127adb6e9890979df2" exitCode=0
Sep 17 11:21:08.170516 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:08.170333    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"
Sep 17 11:21:08.170516 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:08.170348    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9" event={"ID":"169c2f46-9f99-48c4-b90c-1839dc5c2fc1","Type":"ContainerDied","Data":"3e79b6c4e83fb4872e0ba06da28ebf4f94021f17ecca92127adb6e9890979df2"}
Sep 17 11:21:08.170516 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:08.170401    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9" event={"ID":"169c2f46-9f99-48c4-b90c-1839dc5c2fc1","Type":"ContainerDied","Data":"d628f0ebe12becd25a7fcbaa36123e5e52e2fab7900e1541369963a131c075ec"}
Sep 17 11:21:08.170516 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:08.170432    2562 scope.go:117] "RemoveContainer" containerID="3e79b6c4e83fb4872e0ba06da28ebf4f94021f17ecca92127adb6e9890979df2"
Sep 17 11:21:08.179838 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:08.179802    2562 scope.go:117] "RemoveContainer" containerID="3e79b6c4e83fb4872e0ba06da28ebf4f94021f17ecca92127adb6e9890979df2"
Sep 17 11:21:08.180120 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:21:08.180099    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e79b6c4e83fb4872e0ba06da28ebf4f94021f17ecca92127adb6e9890979df2\": container with ID starting with 3e79b6c4e83fb4872e0ba06da28ebf4f94021f17ecca92127adb6e9890979df2 not found: ID does not exist" containerID="3e79b6c4e83fb4872e0ba06da28ebf4f94021f17ecca92127adb6e9890979df2"
Sep 17 11:21:08.180190 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:08.180128    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e79b6c4e83fb4872e0ba06da28ebf4f94021f17ecca92127adb6e9890979df2"} err="failed to get container status \"3e79b6c4e83fb4872e0ba06da28ebf4f94021f17ecca92127adb6e9890979df2\": rpc error: code = NotFound desc = could not find container \"3e79b6c4e83fb4872e0ba06da28ebf4f94021f17ecca92127adb6e9890979df2\": container with ID starting with 3e79b6c4e83fb4872e0ba06da28ebf4f94021f17ecca92127adb6e9890979df2 not found: ID does not exist"
Sep 17 11:21:08.183097 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:08.183077    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:08.192735 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:08.192711    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"]
Sep 17 11:21:08.197277 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:08.197253    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9f97d7-8cfb8f8df-t2dv9"]
Sep 17 11:21:08.422085 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:08.422007    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:08.455595 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:08.455550    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="169c2f46-9f99-48c4-b90c-1839dc5c2fc1" path="/var/lib/kubelet/pods/169c2f46-9f99-48c4-b90c-1839dc5c2fc1/volumes"
Sep 17 11:21:09.402839 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:09.402779    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:10.479852 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:10.479804    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:11.587667 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:11.587636    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:12.390608 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:12.390577    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:13.307958 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:13.307923    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:14.697026 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:14.696992    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:14.797110 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:14.797079    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:14.897548 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:14.897519    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:14.977478 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:14.977400    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:15.927946 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:15.927908    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:21:15.950523 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:15.950491    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:16.740853 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:16.740822    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:17.765124 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:17.765087    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:18.682452 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:18.682415    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:19.547892 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:19.547859    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:19.658991 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:19.658959    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:20.583486 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:20.583456    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:25.545257 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:25.545222    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:26.150517 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:26.150487    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:26.224344 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:26.224291    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:26.325578 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:26.325547    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:26.426025 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:26.425940    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:26.749795 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:26.749716    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:27.084808 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:27.084780    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:27.672023 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:27.671991    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:27.826417 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:27.826380    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:28.658879 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:28.658849    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:29.510179 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:29.510150    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:30.424828 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:30.424772    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:30.736014 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:30.735927    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:21:33.413903 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:33.413869    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:33.520247 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:33.520219    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:33.691109 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:33.691028    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:35.290381 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:35.290344    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:35.960448 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:35.960405    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:36.713265 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:36.713221    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:37.587423 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:37.587390    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:38.682705 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:38.682665    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:44.520447 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.520408    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"]
Sep 17 11:21:44.520875 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.520680    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r" podUID="ac0bdb7b-c601-4d7a-84a9-936f1e228b34" containerName="maas-api" containerID="cri-o://f03de38c42286584147a6927b957768b0e1fe6e2ba2ce0fba53538dcbe5a5e3f" gracePeriod=30
Sep 17 11:21:44.555162 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.555132    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:44.634938 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.634910    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:44.747000 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.746967    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:44.776980 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.776959    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:21:44.789924 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.789897    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-openshift-service-ca\") pod \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\" (UID: \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\") "
Sep 17 11:21:44.790023 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.789933    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2lj6\" (UniqueName: \"kubernetes.io/projected/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-kube-api-access-z2lj6\") pod \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\" (UID: \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\") "
Sep 17 11:21:44.790023 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.789985    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-metrics-certs\") pod \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\" (UID: \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\") "
Sep 17 11:21:44.790103 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.790053    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-maas-api-tls\") pod \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\" (UID: \"ac0bdb7b-c601-4d7a-84a9-936f1e228b34\") "
Sep 17 11:21:44.790298 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.790278    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ac0bdb7b-c601-4d7a-84a9-936f1e228b34" (UID: "ac0bdb7b-c601-4d7a-84a9-936f1e228b34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:21:44.790381 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.790348    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:21:44.792117 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.792092    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ac0bdb7b-c601-4d7a-84a9-936f1e228b34" (UID: "ac0bdb7b-c601-4d7a-84a9-936f1e228b34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:21:44.792117 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.792106    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-kube-api-access-z2lj6" (OuterVolumeSpecName: "kube-api-access-z2lj6") pod "ac0bdb7b-c601-4d7a-84a9-936f1e228b34" (UID: "ac0bdb7b-c601-4d7a-84a9-936f1e228b34"). InnerVolumeSpecName "kube-api-access-z2lj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:21:44.792225 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.792102    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ac0bdb7b-c601-4d7a-84a9-936f1e228b34" (UID: "ac0bdb7b-c601-4d7a-84a9-936f1e228b34"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:21:44.884774 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.884741    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:44.891750 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.891731    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:21:44.891863 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.891752    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2lj6\" (UniqueName: \"kubernetes.io/projected/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-kube-api-access-z2lj6\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:21:44.891863 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.891762    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac0bdb7b-c601-4d7a-84a9-936f1e228b34-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:21:44.960148 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:44.960113    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:45.230336 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:45.230254    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:45.363900 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:45.363860    2562 generic.go:356] "Generic (PLEG): container finished" podID="ac0bdb7b-c601-4d7a-84a9-936f1e228b34" containerID="f03de38c42286584147a6927b957768b0e1fe6e2ba2ce0fba53538dcbe5a5e3f" exitCode=0
Sep 17 11:21:45.363900 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:45.363905    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"
Sep 17 11:21:45.364147 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:45.363941    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r" event={"ID":"ac0bdb7b-c601-4d7a-84a9-936f1e228b34","Type":"ContainerDied","Data":"f03de38c42286584147a6927b957768b0e1fe6e2ba2ce0fba53538dcbe5a5e3f"}
Sep 17 11:21:45.364147 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:45.364006    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r" event={"ID":"ac0bdb7b-c601-4d7a-84a9-936f1e228b34","Type":"ContainerDied","Data":"491a13668950c7192a7b92878f5f6751dd7102a36299b8ee0226ab3f7339c6e5"}
Sep 17 11:21:45.364147 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:45.364055    2562 scope.go:117] "RemoveContainer" containerID="f03de38c42286584147a6927b957768b0e1fe6e2ba2ce0fba53538dcbe5a5e3f"
Sep 17 11:21:45.373587 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:45.373571    2562 scope.go:117] "RemoveContainer" containerID="f03de38c42286584147a6927b957768b0e1fe6e2ba2ce0fba53538dcbe5a5e3f"
Sep 17 11:21:45.373881 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:21:45.373854    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f03de38c42286584147a6927b957768b0e1fe6e2ba2ce0fba53538dcbe5a5e3f\": container with ID starting with f03de38c42286584147a6927b957768b0e1fe6e2ba2ce0fba53538dcbe5a5e3f not found: ID does not exist" containerID="f03de38c42286584147a6927b957768b0e1fe6e2ba2ce0fba53538dcbe5a5e3f"
Sep 17 11:21:45.373958 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:45.373888    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f03de38c42286584147a6927b957768b0e1fe6e2ba2ce0fba53538dcbe5a5e3f"} err="failed to get container status \"f03de38c42286584147a6927b957768b0e1fe6e2ba2ce0fba53538dcbe5a5e3f\": rpc error: code = NotFound desc = could not find container \"f03de38c42286584147a6927b957768b0e1fe6e2ba2ce0fba53538dcbe5a5e3f\": container with ID starting with f03de38c42286584147a6927b957768b0e1fe6e2ba2ce0fba53538dcbe5a5e3f not found: ID does not exist"
Sep 17 11:21:45.386475 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:45.386453    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"]
Sep 17 11:21:45.392710 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:45.392691    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947f83-9ff77f594-65x6r"]
Sep 17 11:21:45.514269 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:45.514183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:46.244737 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:46.244703    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:46.456186 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:46.456145    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac0bdb7b-c601-4d7a-84a9-936f1e228b34" path="/var/lib/kubelet/pods/ac0bdb7b-c601-4d7a-84a9-936f1e228b34/volumes"
Sep 17 11:21:47.133158 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:47.133122    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:48.063976 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:48.063939    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:48.814759 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:48.814719    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:21:51.177147 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:51.177107    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:51.277312 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:51.277275    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:51.517908 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:51.517798    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:57.317996 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.317961    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"]
Sep 17 11:21:57.318402 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.318346    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="169c2f46-9f99-48c4-b90c-1839dc5c2fc1" containerName="maas-api"
Sep 17 11:21:57.318402 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.318356    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="169c2f46-9f99-48c4-b90c-1839dc5c2fc1" containerName="maas-api"
Sep 17 11:21:57.318402 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.318371    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac0bdb7b-c601-4d7a-84a9-936f1e228b34" containerName="maas-api"
Sep 17 11:21:57.318402 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.318377    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac0bdb7b-c601-4d7a-84a9-936f1e228b34" containerName="maas-api"
Sep 17 11:21:57.318539 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.318449    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="169c2f46-9f99-48c4-b90c-1839dc5c2fc1" containerName="maas-api"
Sep 17 11:21:57.318539 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.318459    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac0bdb7b-c601-4d7a-84a9-936f1e228b34" containerName="maas-api"
Sep 17 11:21:57.322849 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.322807    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.327227 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.327201    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-31fca0\"/\"ipp-route-31fca0-kserve-self-signed-certs\""
Sep 17 11:21:57.327227 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.327203    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-31fca0\"/\"kube-root-ca.crt\""
Sep 17 11:21:57.327404 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.327203    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-31fca0\"/\"default-dockercfg-hcdnq\""
Sep 17 11:21:57.327404 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.327250    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-31fca0\"/\"openshift-service-ca.crt\""
Sep 17 11:21:57.333371 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.333345    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"]
Sep 17 11:21:57.398642 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.398612    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-tmp-dir\") pod \"ipp-route-31fca0-kserve-c9c6845cf-c6x8n\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") " pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.398642 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.398646    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ea2871-4479-41bb-bfda-170b7fe36305-tls-certs\") pod \"ipp-route-31fca0-kserve-c9c6845cf-c6x8n\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") " pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.398916 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.398687    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pfrb\" (UniqueName: \"kubernetes.io/projected/40ea2871-4479-41bb-bfda-170b7fe36305-kube-api-access-4pfrb\") pod \"ipp-route-31fca0-kserve-c9c6845cf-c6x8n\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") " pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.398916 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.398708    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-home\") pod \"ipp-route-31fca0-kserve-c9c6845cf-c6x8n\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") " pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.398916 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.398792    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-model-cache\") pod \"ipp-route-31fca0-kserve-c9c6845cf-c6x8n\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") " pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.398916 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.398895    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-dshm\") pod \"ipp-route-31fca0-kserve-c9c6845cf-c6x8n\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") " pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.429335 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.429302    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:21:57.499844 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.499780    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pfrb\" (UniqueName: \"kubernetes.io/projected/40ea2871-4479-41bb-bfda-170b7fe36305-kube-api-access-4pfrb\") pod \"ipp-route-31fca0-kserve-c9c6845cf-c6x8n\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") " pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.499844 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.499841    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-home\") pod \"ipp-route-31fca0-kserve-c9c6845cf-c6x8n\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") " pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.500117 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.499929    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-model-cache\") pod \"ipp-route-31fca0-kserve-c9c6845cf-c6x8n\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") " pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.500117 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.499985    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-dshm\") pod \"ipp-route-31fca0-kserve-c9c6845cf-c6x8n\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") " pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.500117 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.500020    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-tmp-dir\") pod \"ipp-route-31fca0-kserve-c9c6845cf-c6x8n\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") " pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.500117 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.500043    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ea2871-4479-41bb-bfda-170b7fe36305-tls-certs\") pod \"ipp-route-31fca0-kserve-c9c6845cf-c6x8n\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") " pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.500353 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.500329    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-home\") pod \"ipp-route-31fca0-kserve-c9c6845cf-c6x8n\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") " pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.500440 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.500418    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-model-cache\") pod \"ipp-route-31fca0-kserve-c9c6845cf-c6x8n\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") " pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.500482 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.500439    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-tmp-dir\") pod \"ipp-route-31fca0-kserve-c9c6845cf-c6x8n\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") " pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.502298 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.502280    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-dshm\") pod \"ipp-route-31fca0-kserve-c9c6845cf-c6x8n\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") " pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.502481 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.502464    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ea2871-4479-41bb-bfda-170b7fe36305-tls-certs\") pod \"ipp-route-31fca0-kserve-c9c6845cf-c6x8n\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") " pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.508736 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.508715    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pfrb\" (UniqueName: \"kubernetes.io/projected/40ea2871-4479-41bb-bfda-170b7fe36305-kube-api-access-4pfrb\") pod \"ipp-route-31fca0-kserve-c9c6845cf-c6x8n\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") " pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.632767 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.632676    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:21:57.906917 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:57.906642    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"]
Sep 17 11:21:57.909360 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:21:57.909321    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40ea2871_4479_41bb_bfda_170b7fe36305.slice/crio-d43f420a32a9dd443b640ee11b1e86b88ade7f4e629946339f23c27b7db0f778 WatchSource:0}: Error finding container d43f420a32a9dd443b640ee11b1e86b88ade7f4e629946339f23c27b7db0f778: Status 404 returned error can't find the container with id d43f420a32a9dd443b640ee11b1e86b88ade7f4e629946339f23c27b7db0f778
Sep 17 11:21:58.431966 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:58.431930    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n" event={"ID":"40ea2871-4479-41bb-bfda-170b7fe36305","Type":"ContainerStarted","Data":"1c87b7bec0c62b8ce538fc4d88d992e4e8706e2e8fb15bc47554eda65ea75681"}
Sep 17 11:21:58.431966 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:58.431972    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n" event={"ID":"40ea2871-4479-41bb-bfda-170b7fe36305","Type":"ContainerStarted","Data":"d43f420a32a9dd443b640ee11b1e86b88ade7f4e629946339f23c27b7db0f778"}
Sep 17 11:21:58.458233 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:21:58.458172    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n" podStartSLOduration=1.458152308 podStartE2EDuration="1.458152308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:21:58.456771679 +0000 UTC m=+1712.554817245" watchObservedRunningTime="2026-09-17 11:21:58.458152308 +0000 UTC m=+1712.556197875"
Sep 17 11:22:01.423389 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:01.423351    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:22:07.633391 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:07.633352    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:22:07.633893 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:07.633405    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:22:07.646045 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:07.646019    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:22:08.494140 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:08.494105    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:22:15.214059 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:15.214026    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:22:17.530179 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:17.530141    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:21.303075 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:21.303041    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:21.657446 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:21.657359    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:22.030173 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:22.030145    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:24.450482 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:24.450447    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:25.064785 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:25.064754    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:25.174873 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:25.174841    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:25.390793 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:25.390695    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:26.369519 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:26.369487    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:27.336755 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:27.336724    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:28.051555 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.051525    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"]
Sep 17 11:22:28.057010 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.056981    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:22:28.059681 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.059652    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-fd3842-serving-cert\""
Sep 17 11:22:28.072306 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.072281    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"]
Sep 17 11:22:28.099515 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.099484    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:28.183511 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.183474    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11bb3084-8f56-4ed1-ba66-97825bdc6258-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf\" (UID: \"11bb3084-8f56-4ed1-ba66-97825bdc6258\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:22:28.183511 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.183516    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11bb3084-8f56-4ed1-ba66-97825bdc6258-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf\" (UID: \"11bb3084-8f56-4ed1-ba66-97825bdc6258\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:22:28.183767 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.183589    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sj99\" (UniqueName: \"kubernetes.io/projected/11bb3084-8f56-4ed1-ba66-97825bdc6258-kube-api-access-8sj99\") pod \"maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf\" (UID: \"11bb3084-8f56-4ed1-ba66-97825bdc6258\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:22:28.183767 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.183634    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11bb3084-8f56-4ed1-ba66-97825bdc6258-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf\" (UID: \"11bb3084-8f56-4ed1-ba66-97825bdc6258\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:22:28.201299 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.201264    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:28.284232 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.284199    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11bb3084-8f56-4ed1-ba66-97825bdc6258-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf\" (UID: \"11bb3084-8f56-4ed1-ba66-97825bdc6258\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:22:28.284403 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.284240    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11bb3084-8f56-4ed1-ba66-97825bdc6258-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf\" (UID: \"11bb3084-8f56-4ed1-ba66-97825bdc6258\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:22:28.284403 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.284291    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sj99\" (UniqueName: \"kubernetes.io/projected/11bb3084-8f56-4ed1-ba66-97825bdc6258-kube-api-access-8sj99\") pod \"maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf\" (UID: \"11bb3084-8f56-4ed1-ba66-97825bdc6258\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:22:28.284403 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.284316    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11bb3084-8f56-4ed1-ba66-97825bdc6258-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf\" (UID: \"11bb3084-8f56-4ed1-ba66-97825bdc6258\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:22:28.284642 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:22:28.284440    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-serving-cert: secret "maas-api-e2e-ipp-cleanup-fd3842-serving-cert" not found
Sep 17 11:22:28.284642 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:22:28.284520    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11bb3084-8f56-4ed1-ba66-97825bdc6258-maas-api-tls podName:11bb3084-8f56-4ed1-ba66-97825bdc6258 nodeName:}" failed. No retries permitted until 2026-09-17 11:22:28.78449892 +0000 UTC m=+1742.882544483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11bb3084-8f56-4ed1-ba66-97825bdc6258-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf" (UID: "11bb3084-8f56-4ed1-ba66-97825bdc6258") : secret "maas-api-e2e-ipp-cleanup-fd3842-serving-cert" not found
Sep 17 11:22:28.285024 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.285006    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11bb3084-8f56-4ed1-ba66-97825bdc6258-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf\" (UID: \"11bb3084-8f56-4ed1-ba66-97825bdc6258\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:22:28.286840 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.286803    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11bb3084-8f56-4ed1-ba66-97825bdc6258-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf\" (UID: \"11bb3084-8f56-4ed1-ba66-97825bdc6258\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:22:28.297180 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.297155    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sj99\" (UniqueName: \"kubernetes.io/projected/11bb3084-8f56-4ed1-ba66-97825bdc6258-kube-api-access-8sj99\") pod \"maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf\" (UID: \"11bb3084-8f56-4ed1-ba66-97825bdc6258\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:22:28.656527 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.656491    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:28.726157 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.726122    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:28.789865 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.789824    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11bb3084-8f56-4ed1-ba66-97825bdc6258-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf\" (UID: \"11bb3084-8f56-4ed1-ba66-97825bdc6258\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:22:28.792130 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.792108    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11bb3084-8f56-4ed1-ba66-97825bdc6258-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf\" (UID: \"11bb3084-8f56-4ed1-ba66-97825bdc6258\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:22:28.967470 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:28.967374    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:22:29.240438 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:29.240416    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"]
Sep 17 11:22:29.242275 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:22:29.242240    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11bb3084_8f56_4ed1_ba66_97825bdc6258.slice/crio-df64473074bad487dc6485f89004bb2123035d973feff23017d7cfcf0cd815e7 WatchSource:0}: Error finding container df64473074bad487dc6485f89004bb2123035d973feff23017d7cfcf0cd815e7: Status 404 returned error can't find the container with id df64473074bad487dc6485f89004bb2123035d973feff23017d7cfcf0cd815e7
Sep 17 11:22:29.243755 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:29.243735    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 11:22:29.260490 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:29.260470    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:29.588478 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:29.588449    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf" event={"ID":"11bb3084-8f56-4ed1-ba66-97825bdc6258","Type":"ContainerStarted","Data":"df64473074bad487dc6485f89004bb2123035d973feff23017d7cfcf0cd815e7"}
Sep 17 11:22:30.064052 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:30.064019    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:30.594766 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:30.594727    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf" event={"ID":"11bb3084-8f56-4ed1-ba66-97825bdc6258","Type":"ContainerStarted","Data":"84ff976e376d988d3af6cbe16c966dbf4d2715956134cc83979f2b29c3f78024"}
Sep 17 11:22:30.595257 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:30.594879    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:22:30.616164 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:30.616116    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf" podStartSLOduration=2.293818198 podStartE2EDuration="2.616101469s" totalImagesPullingTime="322.283271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:22:30.613793681 +0000 UTC m=+1744.711839247" watchObservedRunningTime="2026-09-17 11:22:30.616101469 +0000 UTC m=+1744.714147036"
Sep 17 11:22:30.864528 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:30.864445    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:36.020585 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:36.020548    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:22:36.605914 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:36.605887    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:22:43.185757 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:43.185723    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:49.034093 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:49.034055    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:50.475938 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:50.475902    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:51.355506 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:51.355466    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:52.048606 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:52.048573    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:52.787083 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:52.787049    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:54.114806 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:54.114778    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:55.131345 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:55.131314    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:56.951498 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:56.951462    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:57.951406 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:57.951373    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:58.681440 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:58.681396    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:22:59.383919 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:22:59.383882    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:00.416570 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:00.416538    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:06.844894 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:06.844862    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"]
Sep 17 11:23:06.845310 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:06.845080    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf" podUID="11bb3084-8f56-4ed1-ba66-97825bdc6258" containerName="maas-api" containerID="cri-o://84ff976e376d988d3af6cbe16c966dbf4d2715956134cc83979f2b29c3f78024" gracePeriod=30
Sep 17 11:23:06.876920 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:06.876874    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:06.960454 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:06.960419    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:07.060962 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.060925    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:07.089715 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.089681    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:23:07.161575 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.161486    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:07.258244 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.258212    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11bb3084-8f56-4ed1-ba66-97825bdc6258-metrics-certs\") pod \"11bb3084-8f56-4ed1-ba66-97825bdc6258\" (UID: \"11bb3084-8f56-4ed1-ba66-97825bdc6258\") "
Sep 17 11:23:07.258422 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.258263    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11bb3084-8f56-4ed1-ba66-97825bdc6258-openshift-service-ca\") pod \"11bb3084-8f56-4ed1-ba66-97825bdc6258\" (UID: \"11bb3084-8f56-4ed1-ba66-97825bdc6258\") "
Sep 17 11:23:07.258422 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.258327    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11bb3084-8f56-4ed1-ba66-97825bdc6258-maas-api-tls\") pod \"11bb3084-8f56-4ed1-ba66-97825bdc6258\" (UID: \"11bb3084-8f56-4ed1-ba66-97825bdc6258\") "
Sep 17 11:23:07.258502 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.258421    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sj99\" (UniqueName: \"kubernetes.io/projected/11bb3084-8f56-4ed1-ba66-97825bdc6258-kube-api-access-8sj99\") pod \"11bb3084-8f56-4ed1-ba66-97825bdc6258\" (UID: \"11bb3084-8f56-4ed1-ba66-97825bdc6258\") "
Sep 17 11:23:07.258753 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.258726    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11bb3084-8f56-4ed1-ba66-97825bdc6258-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "11bb3084-8f56-4ed1-ba66-97825bdc6258" (UID: "11bb3084-8f56-4ed1-ba66-97825bdc6258"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:23:07.260539 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.260514    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11bb3084-8f56-4ed1-ba66-97825bdc6258-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "11bb3084-8f56-4ed1-ba66-97825bdc6258" (UID: "11bb3084-8f56-4ed1-ba66-97825bdc6258"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:23:07.260539 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.260524    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11bb3084-8f56-4ed1-ba66-97825bdc6258-kube-api-access-8sj99" (OuterVolumeSpecName: "kube-api-access-8sj99") pod "11bb3084-8f56-4ed1-ba66-97825bdc6258" (UID: "11bb3084-8f56-4ed1-ba66-97825bdc6258"). InnerVolumeSpecName "kube-api-access-8sj99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:23:07.260729 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.260556    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11bb3084-8f56-4ed1-ba66-97825bdc6258-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "11bb3084-8f56-4ed1-ba66-97825bdc6258" (UID: "11bb3084-8f56-4ed1-ba66-97825bdc6258"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:23:07.359684 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.359634    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11bb3084-8f56-4ed1-ba66-97825bdc6258-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:23:07.359684 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.359675    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11bb3084-8f56-4ed1-ba66-97825bdc6258-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:23:07.359684 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.359686    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11bb3084-8f56-4ed1-ba66-97825bdc6258-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:23:07.359684 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.359695    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sj99\" (UniqueName: \"kubernetes.io/projected/11bb3084-8f56-4ed1-ba66-97825bdc6258-kube-api-access-8sj99\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:23:07.494370 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.494280    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:07.589079 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.589043    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:07.796875 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.796846    2562 generic.go:356] "Generic (PLEG): container finished" podID="11bb3084-8f56-4ed1-ba66-97825bdc6258" containerID="84ff976e376d988d3af6cbe16c966dbf4d2715956134cc83979f2b29c3f78024" exitCode=0
Sep 17 11:23:07.797045 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.796945    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf" event={"ID":"11bb3084-8f56-4ed1-ba66-97825bdc6258","Type":"ContainerDied","Data":"84ff976e376d988d3af6cbe16c966dbf4d2715956134cc83979f2b29c3f78024"}
Sep 17 11:23:07.797045 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.796988    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf" event={"ID":"11bb3084-8f56-4ed1-ba66-97825bdc6258","Type":"ContainerDied","Data":"df64473074bad487dc6485f89004bb2123035d973feff23017d7cfcf0cd815e7"}
Sep 17 11:23:07.797045 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.797009    2562 scope.go:117] "RemoveContainer" containerID="84ff976e376d988d3af6cbe16c966dbf4d2715956134cc83979f2b29c3f78024"
Sep 17 11:23:07.797045 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.796950    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"
Sep 17 11:23:07.808088 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.808065    2562 scope.go:117] "RemoveContainer" containerID="84ff976e376d988d3af6cbe16c966dbf4d2715956134cc83979f2b29c3f78024"
Sep 17 11:23:07.808515 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:23:07.808492    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ff976e376d988d3af6cbe16c966dbf4d2715956134cc83979f2b29c3f78024\": container with ID starting with 84ff976e376d988d3af6cbe16c966dbf4d2715956134cc83979f2b29c3f78024 not found: ID does not exist" containerID="84ff976e376d988d3af6cbe16c966dbf4d2715956134cc83979f2b29c3f78024"
Sep 17 11:23:07.808593 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.808526    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ff976e376d988d3af6cbe16c966dbf4d2715956134cc83979f2b29c3f78024"} err="failed to get container status \"84ff976e376d988d3af6cbe16c966dbf4d2715956134cc83979f2b29c3f78024\": rpc error: code = NotFound desc = could not find container \"84ff976e376d988d3af6cbe16c966dbf4d2715956134cc83979f2b29c3f78024\": container with ID starting with 84ff976e376d988d3af6cbe16c966dbf4d2715956134cc83979f2b29c3f78024 not found: ID does not exist"
Sep 17 11:23:07.821924 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.821895    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"]
Sep 17 11:23:07.827627 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:07.827605    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3842-65f7ddb8ff-mggcf"]
Sep 17 11:23:08.463263 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:08.463220    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11bb3084-8f56-4ed1-ba66-97825bdc6258" path="/var/lib/kubelet/pods/11bb3084-8f56-4ed1-ba66-97825bdc6258/volumes"
Sep 17 11:23:08.555802 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:08.555771    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:09.284210 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:09.284177    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:10.349029 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:10.349001    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:11.520887 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:11.520852    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:23:14.026922 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:14.026892    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:14.225747 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:14.225712    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:14.357678 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:14.357600    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:18.319219 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:18.319185    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:19.269252 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:19.269176    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:20.057888 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:20.057856    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:20.843294 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:20.843246    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:21.567639 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:21.567608    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:26.524515 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:26.524482    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 11:23:26.529315 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:26.529293    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 11:23:31.421126 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:31.421090    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:23:33.663376 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:33.663338    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"]
Sep 17 11:23:33.663747 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:33.663543    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs" podUID="d5c3fc04-6aba-4eaf-9016-f97c6f627fcd" containerName="maas-api" containerID="cri-o://7011dfe270c7e94fdedadd9b7a112d627e5b81ee991c03f95df8f5820696c2f5" gracePeriod=30
Sep 17 11:23:33.702652 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:33.702619    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:33.859621 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:33.859593    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:33.922722 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:33.922665    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:23:33.931010 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:33.930981    2562 generic.go:356] "Generic (PLEG): container finished" podID="d5c3fc04-6aba-4eaf-9016-f97c6f627fcd" containerID="7011dfe270c7e94fdedadd9b7a112d627e5b81ee991c03f95df8f5820696c2f5" exitCode=0
Sep 17 11:23:33.931139 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:33.931018    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs" event={"ID":"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd","Type":"ContainerDied","Data":"7011dfe270c7e94fdedadd9b7a112d627e5b81ee991c03f95df8f5820696c2f5"}
Sep 17 11:23:33.931139 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:33.931025    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"
Sep 17 11:23:33.931139 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:33.931056    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs" event={"ID":"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd","Type":"ContainerDied","Data":"e4fa9d1f3eb078ba39f609051a3a2adc2d28e5b47bea463c014977ee7900f7b4"}
Sep 17 11:23:33.931139 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:33.931076    2562 scope.go:117] "RemoveContainer" containerID="7011dfe270c7e94fdedadd9b7a112d627e5b81ee991c03f95df8f5820696c2f5"
Sep 17 11:23:33.940042 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:33.940026    2562 scope.go:117] "RemoveContainer" containerID="7011dfe270c7e94fdedadd9b7a112d627e5b81ee991c03f95df8f5820696c2f5"
Sep 17 11:23:33.940275 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:23:33.940257    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7011dfe270c7e94fdedadd9b7a112d627e5b81ee991c03f95df8f5820696c2f5\": container with ID starting with 7011dfe270c7e94fdedadd9b7a112d627e5b81ee991c03f95df8f5820696c2f5 not found: ID does not exist" containerID="7011dfe270c7e94fdedadd9b7a112d627e5b81ee991c03f95df8f5820696c2f5"
Sep 17 11:23:33.940331 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:33.940283    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7011dfe270c7e94fdedadd9b7a112d627e5b81ee991c03f95df8f5820696c2f5"} err="failed to get container status \"7011dfe270c7e94fdedadd9b7a112d627e5b81ee991c03f95df8f5820696c2f5\": rpc error: code = NotFound desc = could not find container \"7011dfe270c7e94fdedadd9b7a112d627e5b81ee991c03f95df8f5820696c2f5\": container with ID starting with 7011dfe270c7e94fdedadd9b7a112d627e5b81ee991c03f95df8f5820696c2f5 not found: ID does not exist"
Sep 17 11:23:34.010375 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.010346    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-maas-api-tls\") pod \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\" (UID: \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\") "
Sep 17 11:23:34.010575 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.010444    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-openshift-service-ca\") pod \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\" (UID: \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\") "
Sep 17 11:23:34.010646 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.010579    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kwxj\" (UniqueName: \"kubernetes.io/projected/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-kube-api-access-6kwxj\") pod \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\" (UID: \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\") "
Sep 17 11:23:34.010646 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.010616    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-metrics-certs\") pod \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\" (UID: \"d5c3fc04-6aba-4eaf-9016-f97c6f627fcd\") "
Sep 17 11:23:34.010844 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.010793    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d5c3fc04-6aba-4eaf-9016-f97c6f627fcd" (UID: "d5c3fc04-6aba-4eaf-9016-f97c6f627fcd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:23:34.011030 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.011013    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:23:34.012700 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.012675    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d5c3fc04-6aba-4eaf-9016-f97c6f627fcd" (UID: "d5c3fc04-6aba-4eaf-9016-f97c6f627fcd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:23:34.012700 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.012692    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-kube-api-access-6kwxj" (OuterVolumeSpecName: "kube-api-access-6kwxj") pod "d5c3fc04-6aba-4eaf-9016-f97c6f627fcd" (UID: "d5c3fc04-6aba-4eaf-9016-f97c6f627fcd"). InnerVolumeSpecName "kube-api-access-6kwxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:23:34.012806 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.012745    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d5c3fc04-6aba-4eaf-9016-f97c6f627fcd" (UID: "d5c3fc04-6aba-4eaf-9016-f97c6f627fcd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:23:34.032024 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.032000    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:34.112211 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.112176    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:23:34.112211 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.112201    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:23:34.112211 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.112211    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kwxj\" (UniqueName: \"kubernetes.io/projected/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd-kube-api-access-6kwxj\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:23:34.141315 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.141285    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:34.235343 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.235249    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:34.261711 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.261677    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"]
Sep 17 11:23:34.271780 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.271752    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-31fca0-84d789ff7f-j8tfs"]
Sep 17 11:23:34.463128 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.463088    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5c3fc04-6aba-4eaf-9016-f97c6f627fcd" path="/var/lib/kubelet/pods/d5c3fc04-6aba-4eaf-9016-f97c6f627fcd/volumes"
Sep 17 11:23:34.746053 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:34.746014    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:35.006730 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:35.006628    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:36.538388 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:36.538357    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:37.292195 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:37.292159    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:39.564228 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:39.564194    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"]
Sep 17 11:23:39.657753 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:39.657722    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:23:45.021338 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:45.021302    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:23:54.583735 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:54.583692    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n" podUID="40ea2871-4479-41bb-bfda-170b7fe36305" containerName="main" containerID="cri-o://1c87b7bec0c62b8ce538fc4d88d992e4e8706e2e8fb15bc47554eda65ea75681" gracePeriod=45
Sep 17 11:23:54.830391 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:54.830367    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:23:54.899279 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:54.899185    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-dshm\") pod \"40ea2871-4479-41bb-bfda-170b7fe36305\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") "
Sep 17 11:23:54.899279 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:54.899221    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-model-cache\") pod \"40ea2871-4479-41bb-bfda-170b7fe36305\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") "
Sep 17 11:23:54.899279 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:54.899255    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pfrb\" (UniqueName: \"kubernetes.io/projected/40ea2871-4479-41bb-bfda-170b7fe36305-kube-api-access-4pfrb\") pod \"40ea2871-4479-41bb-bfda-170b7fe36305\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") "
Sep 17 11:23:54.899279 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:54.899272    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ea2871-4479-41bb-bfda-170b7fe36305-tls-certs\") pod \"40ea2871-4479-41bb-bfda-170b7fe36305\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") "
Sep 17 11:23:54.899608 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:54.899303    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-tmp-dir\") pod \"40ea2871-4479-41bb-bfda-170b7fe36305\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") "
Sep 17 11:23:54.899608 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:54.899327    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-home\") pod \"40ea2871-4479-41bb-bfda-170b7fe36305\" (UID: \"40ea2871-4479-41bb-bfda-170b7fe36305\") "
Sep 17 11:23:54.899608 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:54.899562    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40ea2871-4479-41bb-bfda-170b7fe36305" (UID: "40ea2871-4479-41bb-bfda-170b7fe36305"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:23:54.899608 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:54.899580    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-model-cache" (OuterVolumeSpecName: "model-cache") pod "40ea2871-4479-41bb-bfda-170b7fe36305" (UID: "40ea2871-4479-41bb-bfda-170b7fe36305"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:23:54.899608 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:54.899598    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-home" (OuterVolumeSpecName: "home") pod "40ea2871-4479-41bb-bfda-170b7fe36305" (UID: "40ea2871-4479-41bb-bfda-170b7fe36305"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:23:54.901455 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:54.901432    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-dshm" (OuterVolumeSpecName: "dshm") pod "40ea2871-4479-41bb-bfda-170b7fe36305" (UID: "40ea2871-4479-41bb-bfda-170b7fe36305"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:23:54.901635 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:54.901609    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ea2871-4479-41bb-bfda-170b7fe36305-kube-api-access-4pfrb" (OuterVolumeSpecName: "kube-api-access-4pfrb") pod "40ea2871-4479-41bb-bfda-170b7fe36305" (UID: "40ea2871-4479-41bb-bfda-170b7fe36305"). InnerVolumeSpecName "kube-api-access-4pfrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:23:54.901742 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:54.901641    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40ea2871-4479-41bb-bfda-170b7fe36305-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40ea2871-4479-41bb-bfda-170b7fe36305" (UID: "40ea2871-4479-41bb-bfda-170b7fe36305"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:23:55.000447 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:55.000415    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:23:55.000447 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:55.000443    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:23:55.000447 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:55.000454    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pfrb\" (UniqueName: \"kubernetes.io/projected/40ea2871-4479-41bb-bfda-170b7fe36305-kube-api-access-4pfrb\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:23:55.000682 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:55.000464    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ea2871-4479-41bb-bfda-170b7fe36305-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:23:55.000682 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:55.000473    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:23:55.000682 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:55.000480    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ea2871-4479-41bb-bfda-170b7fe36305-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:23:55.041505 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:55.041471    2562 generic.go:356] "Generic (PLEG): container finished" podID="40ea2871-4479-41bb-bfda-170b7fe36305" containerID="1c87b7bec0c62b8ce538fc4d88d992e4e8706e2e8fb15bc47554eda65ea75681" exitCode=0
Sep 17 11:23:55.041667 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:55.041515    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"
Sep 17 11:23:55.041667 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:55.041522    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n" event={"ID":"40ea2871-4479-41bb-bfda-170b7fe36305","Type":"ContainerDied","Data":"1c87b7bec0c62b8ce538fc4d88d992e4e8706e2e8fb15bc47554eda65ea75681"}
Sep 17 11:23:55.041667 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:55.041553    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n" event={"ID":"40ea2871-4479-41bb-bfda-170b7fe36305","Type":"ContainerDied","Data":"d43f420a32a9dd443b640ee11b1e86b88ade7f4e629946339f23c27b7db0f778"}
Sep 17 11:23:55.041667 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:55.041569    2562 scope.go:117] "RemoveContainer" containerID="1c87b7bec0c62b8ce538fc4d88d992e4e8706e2e8fb15bc47554eda65ea75681"
Sep 17 11:23:55.050681 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:55.050662    2562 scope.go:117] "RemoveContainer" containerID="1c87b7bec0c62b8ce538fc4d88d992e4e8706e2e8fb15bc47554eda65ea75681"
Sep 17 11:23:55.050975 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:23:55.050954    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c87b7bec0c62b8ce538fc4d88d992e4e8706e2e8fb15bc47554eda65ea75681\": container with ID starting with 1c87b7bec0c62b8ce538fc4d88d992e4e8706e2e8fb15bc47554eda65ea75681 not found: ID does not exist" containerID="1c87b7bec0c62b8ce538fc4d88d992e4e8706e2e8fb15bc47554eda65ea75681"
Sep 17 11:23:55.051033 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:55.050983    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c87b7bec0c62b8ce538fc4d88d992e4e8706e2e8fb15bc47554eda65ea75681"} err="failed to get container status \"1c87b7bec0c62b8ce538fc4d88d992e4e8706e2e8fb15bc47554eda65ea75681\": rpc error: code = NotFound desc = could not find container \"1c87b7bec0c62b8ce538fc4d88d992e4e8706e2e8fb15bc47554eda65ea75681\": container with ID starting with 1c87b7bec0c62b8ce538fc4d88d992e4e8706e2e8fb15bc47554eda65ea75681 not found: ID does not exist"
Sep 17 11:23:55.062984 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:55.062961    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"]
Sep 17 11:23:55.066370 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:55.066350    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-31fca0/ipp-route-31fca0-kserve-c9c6845cf-c6x8n"]
Sep 17 11:23:56.455705 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:23:56.455663    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40ea2871-4479-41bb-bfda-170b7fe36305" path="/var/lib/kubelet/pods/40ea2871-4479-41bb-bfda-170b7fe36305/volumes"
Sep 17 11:24:06.224332 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:06.224297    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:17.032250 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:17.032211    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:17.373328 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:17.373241    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:24.628912 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:24.628870    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:24.736665 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:24.736638    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:25.081990 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:25.081962    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:25.543562 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:25.543518    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:26.371061 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:26.371033    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:27.479199 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:27.479163    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:28.247666 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:28.247633    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:35.067754 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:35.067715    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:35.989762 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:35.989724    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"]
Sep 17 11:24:35.990096 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:35.990066    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9" podUID="b3e55b40-b624-4fef-b839-973748e993c6" containerName="maas-api" containerID="cri-o://ee2b3ebb16d69087167cb0c7263475d5c323e720ff98d3d41d4b8bb44bb9fe6f" gracePeriod=30
Sep 17 11:24:36.021398 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.021366    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:36.109216 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.109191    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:36.131296 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.131250    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9" podUID="b3e55b40-b624-4fef-b839-973748e993c6" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.76:8443/health\": dial tcp 10.133.0.76:8443: connect: connection refused"
Sep 17 11:24:36.234553 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.234510    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:36.236804 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.236785    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"
Sep 17 11:24:36.248633 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.248565    2562 generic.go:356] "Generic (PLEG): container finished" podID="b3e55b40-b624-4fef-b839-973748e993c6" containerID="ee2b3ebb16d69087167cb0c7263475d5c323e720ff98d3d41d4b8bb44bb9fe6f" exitCode=0
Sep 17 11:24:36.248633 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.248613    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"
Sep 17 11:24:36.248798 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.248661    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9" event={"ID":"b3e55b40-b624-4fef-b839-973748e993c6","Type":"ContainerDied","Data":"ee2b3ebb16d69087167cb0c7263475d5c323e720ff98d3d41d4b8bb44bb9fe6f"}
Sep 17 11:24:36.248798 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.248716    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9" event={"ID":"b3e55b40-b624-4fef-b839-973748e993c6","Type":"ContainerDied","Data":"55e9c1350dc0f06cb29961b04e48b6127efa5a00da224e9465bf891eab9729c9"}
Sep 17 11:24:36.248798 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.248741    2562 scope.go:117] "RemoveContainer" containerID="ee2b3ebb16d69087167cb0c7263475d5c323e720ff98d3d41d4b8bb44bb9fe6f"
Sep 17 11:24:36.258118 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.258088    2562 scope.go:117] "RemoveContainer" containerID="ee2b3ebb16d69087167cb0c7263475d5c323e720ff98d3d41d4b8bb44bb9fe6f"
Sep 17 11:24:36.258457 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:24:36.258422    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee2b3ebb16d69087167cb0c7263475d5c323e720ff98d3d41d4b8bb44bb9fe6f\": container with ID starting with ee2b3ebb16d69087167cb0c7263475d5c323e720ff98d3d41d4b8bb44bb9fe6f not found: ID does not exist" containerID="ee2b3ebb16d69087167cb0c7263475d5c323e720ff98d3d41d4b8bb44bb9fe6f"
Sep 17 11:24:36.258567 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.258464    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2b3ebb16d69087167cb0c7263475d5c323e720ff98d3d41d4b8bb44bb9fe6f"} err="failed to get container status \"ee2b3ebb16d69087167cb0c7263475d5c323e720ff98d3d41d4b8bb44bb9fe6f\": rpc error: code = NotFound desc = could not find container \"ee2b3ebb16d69087167cb0c7263475d5c323e720ff98d3d41d4b8bb44bb9fe6f\": container with ID starting with ee2b3ebb16d69087167cb0c7263475d5c323e720ff98d3d41d4b8bb44bb9fe6f not found: ID does not exist"
Sep 17 11:24:36.297462 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.297434    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e55b40-b624-4fef-b839-973748e993c6-openshift-service-ca\") pod \"b3e55b40-b624-4fef-b839-973748e993c6\" (UID: \"b3e55b40-b624-4fef-b839-973748e993c6\") "
Sep 17 11:24:36.297583 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.297492    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3e55b40-b624-4fef-b839-973748e993c6-maas-api-tls\") pod \"b3e55b40-b624-4fef-b839-973748e993c6\" (UID: \"b3e55b40-b624-4fef-b839-973748e993c6\") "
Sep 17 11:24:36.297583 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.297517    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3e55b40-b624-4fef-b839-973748e993c6-metrics-certs\") pod \"b3e55b40-b624-4fef-b839-973748e993c6\" (UID: \"b3e55b40-b624-4fef-b839-973748e993c6\") "
Sep 17 11:24:36.297583 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.297539    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xltsw\" (UniqueName: \"kubernetes.io/projected/b3e55b40-b624-4fef-b839-973748e993c6-kube-api-access-xltsw\") pod \"b3e55b40-b624-4fef-b839-973748e993c6\" (UID: \"b3e55b40-b624-4fef-b839-973748e993c6\") "
Sep 17 11:24:36.297832 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.297793    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3e55b40-b624-4fef-b839-973748e993c6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b3e55b40-b624-4fef-b839-973748e993c6" (UID: "b3e55b40-b624-4fef-b839-973748e993c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:24:36.299634 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.299602    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e55b40-b624-4fef-b839-973748e993c6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b3e55b40-b624-4fef-b839-973748e993c6" (UID: "b3e55b40-b624-4fef-b839-973748e993c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:24:36.299634 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.299615    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e55b40-b624-4fef-b839-973748e993c6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b3e55b40-b624-4fef-b839-973748e993c6" (UID: "b3e55b40-b624-4fef-b839-973748e993c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:24:36.299831 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.299651    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e55b40-b624-4fef-b839-973748e993c6-kube-api-access-xltsw" (OuterVolumeSpecName: "kube-api-access-xltsw") pod "b3e55b40-b624-4fef-b839-973748e993c6" (UID: "b3e55b40-b624-4fef-b839-973748e993c6"). InnerVolumeSpecName "kube-api-access-xltsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:24:36.332986 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.332956    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:36.398161 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.398129    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3e55b40-b624-4fef-b839-973748e993c6-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:24:36.398161 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.398156    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3e55b40-b624-4fef-b839-973748e993c6-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:24:36.398161 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.398165    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xltsw\" (UniqueName: \"kubernetes.io/projected/b3e55b40-b624-4fef-b839-973748e993c6-kube-api-access-xltsw\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:24:36.398379 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.398175    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e55b40-b624-4fef-b839-973748e993c6-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:24:36.564633 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.564601    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"]
Sep 17 11:24:36.568962 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.568937    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-23a45b-5f9b7d8c6c-5swr9"]
Sep 17 11:24:36.645460 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.645434    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:36.962931 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:36.962850    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:37.858072 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:37.858042    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:38.456005 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:38.455974    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3e55b40-b624-4fef-b839-973748e993c6" path="/var/lib/kubelet/pods/b3e55b40-b624-4fef-b839-973748e993c6/volumes"
Sep 17 11:24:39.513481 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:39.513443    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:24:42.536685 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:42.536649    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:42.637027 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:42.636990    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:42.825908 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:42.825796    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:44.443422 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:44.443386    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:45.207783 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:45.207750    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:53.808552 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:53.808463    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"]
Sep 17 11:24:53.809123 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:53.808748    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r" podUID="9e33cfb7-8379-4eac-bf61-e2294e2563ce" containerName="maas-api" containerID="cri-o://c1dd44b22a7711cec599831ea08482143de1d7e41a7de1bf186df3dac99e37f4" gracePeriod=30
Sep 17 11:24:53.844984 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:53.844915    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:53.951106 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:53.951077    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:54.055680 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.055652    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:54.057098 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.057078    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:24:54.145667 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.145559    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:54.161554 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.161525    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e33cfb7-8379-4eac-bf61-e2294e2563ce-openshift-service-ca\") pod \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\" (UID: \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\") "
Sep 17 11:24:54.161736 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.161604    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e33cfb7-8379-4eac-bf61-e2294e2563ce-metrics-certs\") pod \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\" (UID: \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\") "
Sep 17 11:24:54.161736 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.161641    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpprh\" (UniqueName: \"kubernetes.io/projected/9e33cfb7-8379-4eac-bf61-e2294e2563ce-kube-api-access-tpprh\") pod \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\" (UID: \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\") "
Sep 17 11:24:54.161736 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.161664    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e33cfb7-8379-4eac-bf61-e2294e2563ce-maas-api-tls\") pod \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\" (UID: \"9e33cfb7-8379-4eac-bf61-e2294e2563ce\") "
Sep 17 11:24:54.162008 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.161979    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e33cfb7-8379-4eac-bf61-e2294e2563ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9e33cfb7-8379-4eac-bf61-e2294e2563ce" (UID: "9e33cfb7-8379-4eac-bf61-e2294e2563ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:24:54.163877 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.163850    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e33cfb7-8379-4eac-bf61-e2294e2563ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9e33cfb7-8379-4eac-bf61-e2294e2563ce" (UID: "9e33cfb7-8379-4eac-bf61-e2294e2563ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:24:54.163969 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.163912    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e33cfb7-8379-4eac-bf61-e2294e2563ce-kube-api-access-tpprh" (OuterVolumeSpecName: "kube-api-access-tpprh") pod "9e33cfb7-8379-4eac-bf61-e2294e2563ce" (UID: "9e33cfb7-8379-4eac-bf61-e2294e2563ce"). InnerVolumeSpecName "kube-api-access-tpprh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:24:54.164021 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.164003    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e33cfb7-8379-4eac-bf61-e2294e2563ce-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9e33cfb7-8379-4eac-bf61-e2294e2563ce" (UID: "9e33cfb7-8379-4eac-bf61-e2294e2563ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:24:54.262908 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.262870    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpprh\" (UniqueName: \"kubernetes.io/projected/9e33cfb7-8379-4eac-bf61-e2294e2563ce-kube-api-access-tpprh\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:24:54.262908 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.262904    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e33cfb7-8379-4eac-bf61-e2294e2563ce-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:24:54.262908 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.262913    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e33cfb7-8379-4eac-bf61-e2294e2563ce-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:24:54.263160 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.262923    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e33cfb7-8379-4eac-bf61-e2294e2563ce-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:24:54.340047 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.340015    2562 generic.go:356] "Generic (PLEG): container finished" podID="9e33cfb7-8379-4eac-bf61-e2294e2563ce" containerID="c1dd44b22a7711cec599831ea08482143de1d7e41a7de1bf186df3dac99e37f4" exitCode=0
Sep 17 11:24:54.340198 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.340060    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r" event={"ID":"9e33cfb7-8379-4eac-bf61-e2294e2563ce","Type":"ContainerDied","Data":"c1dd44b22a7711cec599831ea08482143de1d7e41a7de1bf186df3dac99e37f4"}
Sep 17 11:24:54.340198 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.340077    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"
Sep 17 11:24:54.340198 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.340126    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r" event={"ID":"9e33cfb7-8379-4eac-bf61-e2294e2563ce","Type":"ContainerDied","Data":"ee270bc58616d66107725bb8fe1e7a2fbf5ec8dbc5fffb3daf994481a3b7819f"}
Sep 17 11:24:54.340198 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.340147    2562 scope.go:117] "RemoveContainer" containerID="c1dd44b22a7711cec599831ea08482143de1d7e41a7de1bf186df3dac99e37f4"
Sep 17 11:24:54.349419 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.349402    2562 scope.go:117] "RemoveContainer" containerID="c1dd44b22a7711cec599831ea08482143de1d7e41a7de1bf186df3dac99e37f4"
Sep 17 11:24:54.349682 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:24:54.349663    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1dd44b22a7711cec599831ea08482143de1d7e41a7de1bf186df3dac99e37f4\": container with ID starting with c1dd44b22a7711cec599831ea08482143de1d7e41a7de1bf186df3dac99e37f4 not found: ID does not exist" containerID="c1dd44b22a7711cec599831ea08482143de1d7e41a7de1bf186df3dac99e37f4"
Sep 17 11:24:54.349735 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.349691    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1dd44b22a7711cec599831ea08482143de1d7e41a7de1bf186df3dac99e37f4"} err="failed to get container status \"c1dd44b22a7711cec599831ea08482143de1d7e41a7de1bf186df3dac99e37f4\": rpc error: code = NotFound desc = could not find container \"c1dd44b22a7711cec599831ea08482143de1d7e41a7de1bf186df3dac99e37f4\": container with ID starting with c1dd44b22a7711cec599831ea08482143de1d7e41a7de1bf186df3dac99e37f4 not found: ID does not exist"
Sep 17 11:24:54.363153 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.363119    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"]
Sep 17 11:24:54.368173 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.368146    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-026f86-69f7bcb859-krd4r"]
Sep 17 11:24:54.456432 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.456354    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e33cfb7-8379-4eac-bf61-e2294e2563ce" path="/var/lib/kubelet/pods/9e33cfb7-8379-4eac-bf61-e2294e2563ce/volumes"
Sep 17 11:24:54.477930 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.477894    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:54.658393 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:54.658361    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:24:56.718862 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:24:56.718824    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:25:00.700464 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:00.700429    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:25:00.802005 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:00.801967    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:25:00.943793 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:00.943748    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:25:06.662576 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:06.662542    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:25:08.009610 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:08.009565    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:25:08.110784 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:08.110749    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:25:11.782051 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:11.782018    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"]
Sep 17 11:25:11.782469 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:11.782231    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw" podUID="d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41" containerName="maas-api" containerID="cri-o://14fe24e9f4c5341ea28f67ba0b53a07994087e10e969e145148e17ca40876a7c" gracePeriod=30
Sep 17 11:25:11.807435 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:11.807403    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:25:11.878423 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:11.878389    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:25:11.978403 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:11.978375    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:25:12.028171 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.028142    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:25:12.052790 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.052706    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:25:12.129062 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.129027    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-openshift-service-ca\") pod \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\" (UID: \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\") "
Sep 17 11:25:12.129240 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.129094    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8xmz\" (UniqueName: \"kubernetes.io/projected/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-kube-api-access-c8xmz\") pod \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\" (UID: \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\") "
Sep 17 11:25:12.129240 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.129159    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-metrics-certs\") pod \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\" (UID: \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\") "
Sep 17 11:25:12.129240 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.129192    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-maas-api-tls\") pod \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\" (UID: \"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41\") "
Sep 17 11:25:12.129465 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.129443    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41" (UID: "d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 11:25:12.131315 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.131287    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-kube-api-access-c8xmz" (OuterVolumeSpecName: "kube-api-access-c8xmz") pod "d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41" (UID: "d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41"). InnerVolumeSpecName "kube-api-access-c8xmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:25:12.131315 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.131307    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41" (UID: "d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:25:12.131455 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.131292    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41" (UID: "d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:25:12.230356 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.230323    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8xmz\" (UniqueName: \"kubernetes.io/projected/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-kube-api-access-c8xmz\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:25:12.230356 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.230350    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:25:12.230356 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.230360    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:25:12.230356 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.230368    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:25:12.380037 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.379955    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:25:12.431218 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.431186    2562 generic.go:356] "Generic (PLEG): container finished" podID="d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41" containerID="14fe24e9f4c5341ea28f67ba0b53a07994087e10e969e145148e17ca40876a7c" exitCode=0
Sep 17 11:25:12.431403 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.431234    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"
Sep 17 11:25:12.431403 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.431272    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw" event={"ID":"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41","Type":"ContainerDied","Data":"14fe24e9f4c5341ea28f67ba0b53a07994087e10e969e145148e17ca40876a7c"}
Sep 17 11:25:12.431403 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.431324    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw" event={"ID":"d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41","Type":"ContainerDied","Data":"c6c74c3b9d3d272a419033a860fbd1dcc7612b24a9dd59d9b07f50c6b488fb46"}
Sep 17 11:25:12.431403 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.431346    2562 scope.go:117] "RemoveContainer" containerID="14fe24e9f4c5341ea28f67ba0b53a07994087e10e969e145148e17ca40876a7c"
Sep 17 11:25:12.440580 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.440561    2562 scope.go:117] "RemoveContainer" containerID="14fe24e9f4c5341ea28f67ba0b53a07994087e10e969e145148e17ca40876a7c"
Sep 17 11:25:12.440859 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:25:12.440838    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14fe24e9f4c5341ea28f67ba0b53a07994087e10e969e145148e17ca40876a7c\": container with ID starting with 14fe24e9f4c5341ea28f67ba0b53a07994087e10e969e145148e17ca40876a7c not found: ID does not exist" containerID="14fe24e9f4c5341ea28f67ba0b53a07994087e10e969e145148e17ca40876a7c"
Sep 17 11:25:12.440923 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.440870    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14fe24e9f4c5341ea28f67ba0b53a07994087e10e969e145148e17ca40876a7c"} err="failed to get container status \"14fe24e9f4c5341ea28f67ba0b53a07994087e10e969e145148e17ca40876a7c\": rpc error: code = NotFound desc = could not find container \"14fe24e9f4c5341ea28f67ba0b53a07994087e10e969e145148e17ca40876a7c\": container with ID starting with 14fe24e9f4c5341ea28f67ba0b53a07994087e10e969e145148e17ca40876a7c not found: ID does not exist"
Sep 17 11:25:12.467946 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.467916    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"]
Sep 17 11:25:12.489014 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:12.488994    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8287f-765874cb9f-qmtvw"]
Sep 17 11:25:14.455728 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:14.455694    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41" path="/var/lib/kubelet/pods/d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41/volumes"
Sep 17 11:25:14.610192 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:14.610156    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:25:18.764499 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:18.764467    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:25:18.865374 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:18.865334    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:25:19.027220 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:19.027193    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:25:23.414115 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:25:23.414079    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:26:26.722207 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:26:26.722130    2562 scope.go:117] "RemoveContainer" containerID="0b4efb6bc821b996d5228956fc400a49dad0cbf4a29fc5f80245cea144170afb"
Sep 17 11:26:44.918747 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:26:44.918710    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:26:44.937371 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:26:44.937338    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:26:46.525451 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:26:46.525420    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:26:57.574671 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:26:57.574639    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:27:10.617774 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:27:10.617735    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:27:18.913409 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:27:18.913366    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:27:25.910046 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:27:25.910009    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:27:33.514000 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:27:33.513955    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:27:48.615382 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:27:48.615345    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:27:56.191474 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:27:56.191428    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:27:56.625551 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:27:56.625517    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:28:03.465604 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:28:03.465566    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:28:26.561589 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:28:26.561557    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 11:28:26.567587 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:28:26.567567    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 11:29:25.806872 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:25.806778    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:29:32.640898 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:32.640863    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:29:40.919039 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919002    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"]
Sep 17 11:29:40.919433 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919399    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c3fc04-6aba-4eaf-9016-f97c6f627fcd" containerName="maas-api"
Sep 17 11:29:40.919433 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919411    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c3fc04-6aba-4eaf-9016-f97c6f627fcd" containerName="maas-api"
Sep 17 11:29:40.919433 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919424    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40ea2871-4479-41bb-bfda-170b7fe36305" containerName="main"
Sep 17 11:29:40.919433 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919429    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="40ea2871-4479-41bb-bfda-170b7fe36305" containerName="main"
Sep 17 11:29:40.919572 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919442    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e33cfb7-8379-4eac-bf61-e2294e2563ce" containerName="maas-api"
Sep 17 11:29:40.919572 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919448    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e33cfb7-8379-4eac-bf61-e2294e2563ce" containerName="maas-api"
Sep 17 11:29:40.919572 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919454    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3e55b40-b624-4fef-b839-973748e993c6" containerName="maas-api"
Sep 17 11:29:40.919572 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919461    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3e55b40-b624-4fef-b839-973748e993c6" containerName="maas-api"
Sep 17 11:29:40.919572 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919476    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41" containerName="maas-api"
Sep 17 11:29:40.919572 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919482    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41" containerName="maas-api"
Sep 17 11:29:40.919572 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919490    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11bb3084-8f56-4ed1-ba66-97825bdc6258" containerName="maas-api"
Sep 17 11:29:40.919572 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919495    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="11bb3084-8f56-4ed1-ba66-97825bdc6258" containerName="maas-api"
Sep 17 11:29:40.919572 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919560    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e33cfb7-8379-4eac-bf61-e2294e2563ce" containerName="maas-api"
Sep 17 11:29:40.919572 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919569    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d234cffa-5bc1-4fe2-8e62-4f8a9c55bd41" containerName="maas-api"
Sep 17 11:29:40.919572 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919575    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="11bb3084-8f56-4ed1-ba66-97825bdc6258" containerName="maas-api"
Sep 17 11:29:40.919932 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919583    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c3fc04-6aba-4eaf-9016-f97c6f627fcd" containerName="maas-api"
Sep 17 11:29:40.919932 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919590    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3e55b40-b624-4fef-b839-973748e993c6" containerName="maas-api"
Sep 17 11:29:40.919932 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.919598    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="40ea2871-4479-41bb-bfda-170b7fe36305" containerName="main"
Sep 17 11:29:40.922759 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.922743    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:40.925626 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.925599    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-727fa24a-kserve-self-signed-certs\""
Sep 17 11:29:40.931760 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.931732    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"]
Sep 17 11:29:40.977131 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.977099    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:29:40.984355 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.984330    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tqz2\" (UniqueName: \"kubernetes.io/projected/67f4f9cd-8743-4606-b848-b9f23e647dfc-kube-api-access-9tqz2\") pod \"e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") " pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:40.984487 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.984386    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-model-cache\") pod \"e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") " pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:40.984487 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.984457    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-dshm\") pod \"e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") " pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:40.984566 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.984499    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-home\") pod \"e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") " pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:40.984566 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.984525    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-tmp-dir\") pod \"e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") " pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:40.984566 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:40.984539    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f4f9cd-8743-4606-b848-b9f23e647dfc-tls-certs\") pod \"e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") " pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:41.085157 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:41.085122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-dshm\") pod \"e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") " pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:41.085346 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:41.085166    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-home\") pod \"e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") " pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:41.085346 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:41.085186    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-tmp-dir\") pod \"e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") " pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:41.085346 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:41.085274    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f4f9cd-8743-4606-b848-b9f23e647dfc-tls-certs\") pod \"e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") " pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:41.085346 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:41.085304    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tqz2\" (UniqueName: \"kubernetes.io/projected/67f4f9cd-8743-4606-b848-b9f23e647dfc-kube-api-access-9tqz2\") pod \"e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") " pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:41.085346 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:41.085341    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-model-cache\") pod \"e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") " pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:41.085700 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:41.085667    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-tmp-dir\") pod \"e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") " pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:41.085838 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:41.085706    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-home\") pod \"e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") " pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:41.085838 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:41.085735    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-model-cache\") pod \"e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") " pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:41.087530 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:41.087500    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-dshm\") pod \"e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") " pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:41.087770 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:41.087751    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f4f9cd-8743-4606-b848-b9f23e647dfc-tls-certs\") pod \"e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") " pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:41.094373 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:41.094354    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tqz2\" (UniqueName: \"kubernetes.io/projected/67f4f9cd-8743-4606-b848-b9f23e647dfc-kube-api-access-9tqz2\") pod \"e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") " pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:41.232915 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:41.232802    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:41.415249 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:41.415213    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:29:41.507438 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:41.507410    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"]
Sep 17 11:29:41.509714 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:29:41.509687    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67f4f9cd_8743_4606_b848_b9f23e647dfc.slice/crio-7c7382397b59ad5d1cf36fdc0df77f5c0f6a52ff67fae6b0a16f4eaf97221a28 WatchSource:0}: Error finding container 7c7382397b59ad5d1cf36fdc0df77f5c0f6a52ff67fae6b0a16f4eaf97221a28: Status 404 returned error can't find the container with id 7c7382397b59ad5d1cf36fdc0df77f5c0f6a52ff67fae6b0a16f4eaf97221a28
Sep 17 11:29:41.733305 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:41.733263    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr" event={"ID":"67f4f9cd-8743-4606-b848-b9f23e647dfc","Type":"ContainerStarted","Data":"438f07864bc69031892cc6491047197465843b231f9df0856cde558824dee77f"}
Sep 17 11:29:41.733472 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:41.733315    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr" event={"ID":"67f4f9cd-8743-4606-b848-b9f23e647dfc","Type":"ContainerStarted","Data":"7c7382397b59ad5d1cf36fdc0df77f5c0f6a52ff67fae6b0a16f4eaf97221a28"}
Sep 17 11:29:41.751182 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:41.751124    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr" podStartSLOduration=1.7511049669999998 podStartE2EDuration="1.751104967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:29:41.750027073 +0000 UTC m=+2175.848072638" watchObservedRunningTime="2026-09-17 11:29:41.751104967 +0000 UTC m=+2175.849150532"
Sep 17 11:29:48.818381 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:48.818347    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:29:51.232942 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:51.232909    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:51.233353 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:51.232970    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:51.245380 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:51.245352    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:51.802565 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:51.802536    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:29:52.408494 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.408459    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"]
Sep 17 11:29:52.412106 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.412090    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.418182 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.418150    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-727fa24a-kserve-self-signed-certs\""
Sep 17 11:29:52.426244 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.426220    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"]
Sep 17 11:29:52.494178 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.494146    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-dshm\") pod \"e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") " pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.494339 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.494202    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00cbe6fe-bec8-42af-a544-4627ff2343ca-tls-certs\") pod \"e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") " pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.494339 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.494307    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-tmp-dir\") pod \"e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") " pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.494423 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.494345    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-model-cache\") pod \"e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") " pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.494423 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.494369    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg7nv\" (UniqueName: \"kubernetes.io/projected/00cbe6fe-bec8-42af-a544-4627ff2343ca-kube-api-access-tg7nv\") pod \"e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") " pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.494503 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.494433    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-home\") pod \"e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") " pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.527739 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.527698    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:29:52.595213 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.595180    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-tmp-dir\") pod \"e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") " pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.595409 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.595233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-model-cache\") pod \"e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") " pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.595409 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.595261    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg7nv\" (UniqueName: \"kubernetes.io/projected/00cbe6fe-bec8-42af-a544-4627ff2343ca-kube-api-access-tg7nv\") pod \"e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") " pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.595409 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.595304    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-home\") pod \"e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") " pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.595409 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.595327    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-dshm\") pod \"e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") " pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.595409 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.595374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00cbe6fe-bec8-42af-a544-4627ff2343ca-tls-certs\") pod \"e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") " pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.595677 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.595654    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-tmp-dir\") pod \"e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") " pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.595736 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.595704    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-home\") pod \"e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") " pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.595736 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.595706    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-model-cache\") pod \"e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") " pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.597601 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.597574    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-dshm\") pod \"e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") " pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.598025 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.598004    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00cbe6fe-bec8-42af-a544-4627ff2343ca-tls-certs\") pod \"e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") " pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.610046 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.610023    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg7nv\" (UniqueName: \"kubernetes.io/projected/00cbe6fe-bec8-42af-a544-4627ff2343ca-kube-api-access-tg7nv\") pod \"e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") " pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:52.721412 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:52.721325    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:29:53.003304 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:53.003125    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"]
Sep 17 11:29:53.006074 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:29:53.006045    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00cbe6fe_bec8_42af_a544_4627ff2343ca.slice/crio-64253815fa13a73a21cd6b43073e073d2fa8975f2df89fd9fbe4024cc2053247 WatchSource:0}: Error finding container 64253815fa13a73a21cd6b43073e073d2fa8975f2df89fd9fbe4024cc2053247: Status 404 returned error can't find the container with id 64253815fa13a73a21cd6b43073e073d2fa8975f2df89fd9fbe4024cc2053247
Sep 17 11:29:53.803062 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:53.803017    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t" event={"ID":"00cbe6fe-bec8-42af-a544-4627ff2343ca","Type":"ContainerStarted","Data":"6dcfc7c7f47e42f7c6e1cf46bf61172f29bc195d9a5ca5f32db62737f03ef828"}
Sep 17 11:29:53.803430 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:53.803073    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t" event={"ID":"00cbe6fe-bec8-42af-a544-4627ff2343ca","Type":"ContainerStarted","Data":"64253815fa13a73a21cd6b43073e073d2fa8975f2df89fd9fbe4024cc2053247"}
Sep 17 11:29:53.824491 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:53.824441    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t" podStartSLOduration=1.8244254899999999 podStartE2EDuration="1.82442549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:29:53.823394949 +0000 UTC m=+2187.921440515" watchObservedRunningTime="2026-09-17 11:29:53.82442549 +0000 UTC m=+2187.922471057"
Sep 17 11:29:54.513972 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:29:54.513936    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:30:02.722391 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:02.722348    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:30:02.723015 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:02.722406    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:30:02.734855 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:02.734808    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:30:02.862836 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:02.862791    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:30:08.357272 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:08.357237    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"]
Sep 17 11:30:08.381920 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:08.381888    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:30:10.005097 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:10.005064    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"]
Sep 17 11:30:10.392685 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:10.392584    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:30:10.608764 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:10.608729    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:30:19.616806 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:19.616770    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:30:23.377311 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.377274    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr" podUID="67f4f9cd-8743-4606-b848-b9f23e647dfc" containerName="main" containerID="cri-o://438f07864bc69031892cc6491047197465843b231f9df0856cde558824dee77f" gracePeriod=45
Sep 17 11:30:23.624966 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.624940    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:30:23.678545 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.678467    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tqz2\" (UniqueName: \"kubernetes.io/projected/67f4f9cd-8743-4606-b848-b9f23e647dfc-kube-api-access-9tqz2\") pod \"67f4f9cd-8743-4606-b848-b9f23e647dfc\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") "
Sep 17 11:30:23.678545 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.678516    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f4f9cd-8743-4606-b848-b9f23e647dfc-tls-certs\") pod \"67f4f9cd-8743-4606-b848-b9f23e647dfc\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") "
Sep 17 11:30:23.678545 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.678540    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-home\") pod \"67f4f9cd-8743-4606-b848-b9f23e647dfc\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") "
Sep 17 11:30:23.678836 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.678572    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-tmp-dir\") pod \"67f4f9cd-8743-4606-b848-b9f23e647dfc\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") "
Sep 17 11:30:23.678836 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.678608    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-model-cache\") pod \"67f4f9cd-8743-4606-b848-b9f23e647dfc\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") "
Sep 17 11:30:23.678836 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.678638    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-dshm\") pod \"67f4f9cd-8743-4606-b848-b9f23e647dfc\" (UID: \"67f4f9cd-8743-4606-b848-b9f23e647dfc\") "
Sep 17 11:30:23.678836 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.678804    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-home" (OuterVolumeSpecName: "home") pod "67f4f9cd-8743-4606-b848-b9f23e647dfc" (UID: "67f4f9cd-8743-4606-b848-b9f23e647dfc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:30:23.679014 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.678895    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67f4f9cd-8743-4606-b848-b9f23e647dfc" (UID: "67f4f9cd-8743-4606-b848-b9f23e647dfc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:30:23.679014 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.678926    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-model-cache" (OuterVolumeSpecName: "model-cache") pod "67f4f9cd-8743-4606-b848-b9f23e647dfc" (UID: "67f4f9cd-8743-4606-b848-b9f23e647dfc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:30:23.679014 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.678993    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:30:23.679014 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.679007    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:30:23.679149 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.679022    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:30:23.680699 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.680670    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-dshm" (OuterVolumeSpecName: "dshm") pod "67f4f9cd-8743-4606-b848-b9f23e647dfc" (UID: "67f4f9cd-8743-4606-b848-b9f23e647dfc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:30:23.680806 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.680730    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67f4f9cd-8743-4606-b848-b9f23e647dfc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67f4f9cd-8743-4606-b848-b9f23e647dfc" (UID: "67f4f9cd-8743-4606-b848-b9f23e647dfc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:30:23.680806 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.680770    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f4f9cd-8743-4606-b848-b9f23e647dfc-kube-api-access-9tqz2" (OuterVolumeSpecName: "kube-api-access-9tqz2") pod "67f4f9cd-8743-4606-b848-b9f23e647dfc" (UID: "67f4f9cd-8743-4606-b848-b9f23e647dfc"). InnerVolumeSpecName "kube-api-access-9tqz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:30:23.779898 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.779859    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67f4f9cd-8743-4606-b848-b9f23e647dfc-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:30:23.779898 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.779892    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tqz2\" (UniqueName: \"kubernetes.io/projected/67f4f9cd-8743-4606-b848-b9f23e647dfc-kube-api-access-9tqz2\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:30:23.779898 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.779902    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f4f9cd-8743-4606-b848-b9f23e647dfc-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:30:23.950523 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.950437    2562 generic.go:356] "Generic (PLEG): container finished" podID="67f4f9cd-8743-4606-b848-b9f23e647dfc" containerID="438f07864bc69031892cc6491047197465843b231f9df0856cde558824dee77f" exitCode=0
Sep 17 11:30:23.950523 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.950482    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"
Sep 17 11:30:23.950755 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.950528    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr" event={"ID":"67f4f9cd-8743-4606-b848-b9f23e647dfc","Type":"ContainerDied","Data":"438f07864bc69031892cc6491047197465843b231f9df0856cde558824dee77f"}
Sep 17 11:30:23.950755 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.950573    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr" event={"ID":"67f4f9cd-8743-4606-b848-b9f23e647dfc","Type":"ContainerDied","Data":"7c7382397b59ad5d1cf36fdc0df77f5c0f6a52ff67fae6b0a16f4eaf97221a28"}
Sep 17 11:30:23.950755 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.950595    2562 scope.go:117] "RemoveContainer" containerID="438f07864bc69031892cc6491047197465843b231f9df0856cde558824dee77f"
Sep 17 11:30:23.959399 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.959380    2562 scope.go:117] "RemoveContainer" containerID="438f07864bc69031892cc6491047197465843b231f9df0856cde558824dee77f"
Sep 17 11:30:23.959659 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:30:23.959641    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"438f07864bc69031892cc6491047197465843b231f9df0856cde558824dee77f\": container with ID starting with 438f07864bc69031892cc6491047197465843b231f9df0856cde558824dee77f not found: ID does not exist" containerID="438f07864bc69031892cc6491047197465843b231f9df0856cde558824dee77f"
Sep 17 11:30:23.959710 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.959667    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"438f07864bc69031892cc6491047197465843b231f9df0856cde558824dee77f"} err="failed to get container status \"438f07864bc69031892cc6491047197465843b231f9df0856cde558824dee77f\": rpc error: code = NotFound desc = could not find container \"438f07864bc69031892cc6491047197465843b231f9df0856cde558824dee77f\": container with ID starting with 438f07864bc69031892cc6491047197465843b231f9df0856cde558824dee77f not found: ID does not exist"
Sep 17 11:30:23.971968 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.971945    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"]
Sep 17 11:30:23.978098 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:23.978075    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-727fa24a-kserve-748cfc5595-khrbr"]
Sep 17 11:30:24.456229 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:24.456197    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67f4f9cd-8743-4606-b848-b9f23e647dfc" path="/var/lib/kubelet/pods/67f4f9cd-8743-4606-b848-b9f23e647dfc/volumes"
Sep 17 11:30:25.024689 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.024648    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t" podUID="00cbe6fe-bec8-42af-a544-4627ff2343ca" containerName="main" containerID="cri-o://6dcfc7c7f47e42f7c6e1cf46bf61172f29bc195d9a5ca5f32db62737f03ef828" gracePeriod=45
Sep 17 11:30:25.274855 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.274781    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:30:25.293238 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.293219    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg7nv\" (UniqueName: \"kubernetes.io/projected/00cbe6fe-bec8-42af-a544-4627ff2343ca-kube-api-access-tg7nv\") pod \"00cbe6fe-bec8-42af-a544-4627ff2343ca\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") "
Sep 17 11:30:25.293347 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.293276    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-tmp-dir\") pod \"00cbe6fe-bec8-42af-a544-4627ff2343ca\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") "
Sep 17 11:30:25.293347 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.293306    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-home\") pod \"00cbe6fe-bec8-42af-a544-4627ff2343ca\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") "
Sep 17 11:30:25.293415 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.293367    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00cbe6fe-bec8-42af-a544-4627ff2343ca-tls-certs\") pod \"00cbe6fe-bec8-42af-a544-4627ff2343ca\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") "
Sep 17 11:30:25.293463 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.293416    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-dshm\") pod \"00cbe6fe-bec8-42af-a544-4627ff2343ca\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") "
Sep 17 11:30:25.293463 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.293437    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-model-cache\") pod \"00cbe6fe-bec8-42af-a544-4627ff2343ca\" (UID: \"00cbe6fe-bec8-42af-a544-4627ff2343ca\") "
Sep 17 11:30:25.293609 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.293589    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "00cbe6fe-bec8-42af-a544-4627ff2343ca" (UID: "00cbe6fe-bec8-42af-a544-4627ff2343ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:30:25.293694 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.293605    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-home" (OuterVolumeSpecName: "home") pod "00cbe6fe-bec8-42af-a544-4627ff2343ca" (UID: "00cbe6fe-bec8-42af-a544-4627ff2343ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:30:25.293780 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.293756    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "00cbe6fe-bec8-42af-a544-4627ff2343ca" (UID: "00cbe6fe-bec8-42af-a544-4627ff2343ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:30:25.293850 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.293777    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:30:25.293850 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.293801    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:30:25.295322 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.295296    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00cbe6fe-bec8-42af-a544-4627ff2343ca-kube-api-access-tg7nv" (OuterVolumeSpecName: "kube-api-access-tg7nv") pod "00cbe6fe-bec8-42af-a544-4627ff2343ca" (UID: "00cbe6fe-bec8-42af-a544-4627ff2343ca"). InnerVolumeSpecName "kube-api-access-tg7nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 11:30:25.295465 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.295447    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00cbe6fe-bec8-42af-a544-4627ff2343ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00cbe6fe-bec8-42af-a544-4627ff2343ca" (UID: "00cbe6fe-bec8-42af-a544-4627ff2343ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 11:30:25.295530 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.295508    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-dshm" (OuterVolumeSpecName: "dshm") pod "00cbe6fe-bec8-42af-a544-4627ff2343ca" (UID: "00cbe6fe-bec8-42af-a544-4627ff2343ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 11:30:25.394762 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.394730    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00cbe6fe-bec8-42af-a544-4627ff2343ca-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:30:25.394762 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.394759    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:30:25.394762 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.394769    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00cbe6fe-bec8-42af-a544-4627ff2343ca-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:30:25.395012 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.394778    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg7nv\" (UniqueName: \"kubernetes.io/projected/00cbe6fe-bec8-42af-a544-4627ff2343ca-kube-api-access-tg7nv\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\""
Sep 17 11:30:25.962080 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.962035    2562 generic.go:356] "Generic (PLEG): container finished" podID="00cbe6fe-bec8-42af-a544-4627ff2343ca" containerID="6dcfc7c7f47e42f7c6e1cf46bf61172f29bc195d9a5ca5f32db62737f03ef828" exitCode=0
Sep 17 11:30:25.962479 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.962098    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"
Sep 17 11:30:25.962479 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.962115    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t" event={"ID":"00cbe6fe-bec8-42af-a544-4627ff2343ca","Type":"ContainerDied","Data":"6dcfc7c7f47e42f7c6e1cf46bf61172f29bc195d9a5ca5f32db62737f03ef828"}
Sep 17 11:30:25.962479 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.962160    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t" event={"ID":"00cbe6fe-bec8-42af-a544-4627ff2343ca","Type":"ContainerDied","Data":"64253815fa13a73a21cd6b43073e073d2fa8975f2df89fd9fbe4024cc2053247"}
Sep 17 11:30:25.962479 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.962179    2562 scope.go:117] "RemoveContainer" containerID="6dcfc7c7f47e42f7c6e1cf46bf61172f29bc195d9a5ca5f32db62737f03ef828"
Sep 17 11:30:25.971037 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.971023    2562 scope.go:117] "RemoveContainer" containerID="6dcfc7c7f47e42f7c6e1cf46bf61172f29bc195d9a5ca5f32db62737f03ef828"
Sep 17 11:30:25.971306 ip-10-0-139-204 kubenswrapper[2562]: E0917 11:30:25.971287    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dcfc7c7f47e42f7c6e1cf46bf61172f29bc195d9a5ca5f32db62737f03ef828\": container with ID starting with 6dcfc7c7f47e42f7c6e1cf46bf61172f29bc195d9a5ca5f32db62737f03ef828 not found: ID does not exist" containerID="6dcfc7c7f47e42f7c6e1cf46bf61172f29bc195d9a5ca5f32db62737f03ef828"
Sep 17 11:30:25.971351 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.971314    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dcfc7c7f47e42f7c6e1cf46bf61172f29bc195d9a5ca5f32db62737f03ef828"} err="failed to get container status \"6dcfc7c7f47e42f7c6e1cf46bf61172f29bc195d9a5ca5f32db62737f03ef828\": rpc error: code = NotFound desc = could not find container \"6dcfc7c7f47e42f7c6e1cf46bf61172f29bc195d9a5ca5f32db62737f03ef828\": container with ID starting with 6dcfc7c7f47e42f7c6e1cf46bf61172f29bc195d9a5ca5f32db62737f03ef828 not found: ID does not exist"
Sep 17 11:30:25.983689 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.983666    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"]
Sep 17 11:30:25.987779 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:25.987761    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-727fa24a-kserve-594c5b778c-dkp9t"]
Sep 17 11:30:26.456155 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:26.456129    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00cbe6fe-bec8-42af-a544-4627ff2343ca" path="/var/lib/kubelet/pods/00cbe6fe-bec8-42af-a544-4627ff2343ca/volumes"
Sep 17 11:30:29.013017 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:30:29.012982    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:32:52.891375 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:32:52.891285    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:33:26.601183 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:26.601149    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 11:33:26.606546 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:26.606524    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 11:33:35.211141 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:35.211105    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:33:42.019908 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:42.019873    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:33:47.816954 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:47.816917    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:33:53.589950 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:53.589873    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:33:54.814215 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:54.814180    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dr8cp"]
Sep 17 11:33:58.944568 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:58.944527    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tmn9t/must-gather-7tjnr"]
Sep 17 11:33:58.945177 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:58.945024    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f4f9cd-8743-4606-b848-b9f23e647dfc" containerName="main"
Sep 17 11:33:58.945177 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:58.945041    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f4f9cd-8743-4606-b848-b9f23e647dfc" containerName="main"
Sep 17 11:33:58.945177 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:58.945074    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00cbe6fe-bec8-42af-a544-4627ff2343ca" containerName="main"
Sep 17 11:33:58.945177 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:58.945082    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00cbe6fe-bec8-42af-a544-4627ff2343ca" containerName="main"
Sep 17 11:33:58.945177 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:58.945154    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f4f9cd-8743-4606-b848-b9f23e647dfc" containerName="main"
Sep 17 11:33:58.945177 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:58.945163    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="00cbe6fe-bec8-42af-a544-4627ff2343ca" containerName="main"
Sep 17 11:33:58.948369 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:58.948353    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tmn9t/must-gather-7tjnr"
Sep 17 11:33:58.951148 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:58.951124    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tmn9t\"/\"openshift-service-ca.crt\""
Sep 17 11:33:58.951266 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:58.951194    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tmn9t\"/\"kube-root-ca.crt\""
Sep 17 11:33:58.952371 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:58.952349    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tmn9t\"/\"default-dockercfg-scghf\""
Sep 17 11:33:58.953430 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:58.953404    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:33:58.998761 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:58.998721    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tmn9t/must-gather-7tjnr"]
Sep 17 11:33:59.003765 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:59.003738    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dfec70b5-62df-4f31-99d7-e03ace02974a-must-gather-output\") pod \"must-gather-7tjnr\" (UID: \"dfec70b5-62df-4f31-99d7-e03ace02974a\") " pod="openshift-must-gather-tmn9t/must-gather-7tjnr"
Sep 17 11:33:59.003943 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:59.003919    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqsmt\" (UniqueName: \"kubernetes.io/projected/dfec70b5-62df-4f31-99d7-e03ace02974a-kube-api-access-kqsmt\") pod \"must-gather-7tjnr\" (UID: \"dfec70b5-62df-4f31-99d7-e03ace02974a\") " pod="openshift-must-gather-tmn9t/must-gather-7tjnr"
Sep 17 11:33:59.026666 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:59.026633    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log" limit=1024
Sep 17 11:33:59.105096 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:59.105059    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dfec70b5-62df-4f31-99d7-e03ace02974a-must-gather-output\") pod \"must-gather-7tjnr\" (UID: \"dfec70b5-62df-4f31-99d7-e03ace02974a\") " pod="openshift-must-gather-tmn9t/must-gather-7tjnr"
Sep 17 11:33:59.105281 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:59.105159    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqsmt\" (UniqueName: \"kubernetes.io/projected/dfec70b5-62df-4f31-99d7-e03ace02974a-kube-api-access-kqsmt\") pod \"must-gather-7tjnr\" (UID: \"dfec70b5-62df-4f31-99d7-e03ace02974a\") " pod="openshift-must-gather-tmn9t/must-gather-7tjnr"
Sep 17 11:33:59.105434 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:59.105411    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dfec70b5-62df-4f31-99d7-e03ace02974a-must-gather-output\") pod \"must-gather-7tjnr\" (UID: \"dfec70b5-62df-4f31-99d7-e03ace02974a\") " pod="openshift-must-gather-tmn9t/must-gather-7tjnr"
Sep 17 11:33:59.117325 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:59.117297    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqsmt\" (UniqueName: \"kubernetes.io/projected/dfec70b5-62df-4f31-99d7-e03ace02974a-kube-api-access-kqsmt\") pod \"must-gather-7tjnr\" (UID: \"dfec70b5-62df-4f31-99d7-e03ace02974a\") " pod="openshift-must-gather-tmn9t/must-gather-7tjnr"
Sep 17 11:33:59.258420 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:59.258332    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tmn9t/must-gather-7tjnr"
Sep 17 11:33:59.529313 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:59.529289    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tmn9t/must-gather-7tjnr"]
Sep 17 11:33:59.531343 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:33:59.531316    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfec70b5_62df_4f31_99d7_e03ace02974a.slice/crio-d1b52f6c3497955cff7be27a8fd0e5133983b3e4aeb575c0f249d9542aca9d18 WatchSource:0}: Error finding container d1b52f6c3497955cff7be27a8fd0e5133983b3e4aeb575c0f249d9542aca9d18: Status 404 returned error can't find the container with id d1b52f6c3497955cff7be27a8fd0e5133983b3e4aeb575c0f249d9542aca9d18
Sep 17 11:33:59.533155 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:59.533139    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 11:33:59.994400 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:33:59.994321    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tmn9t/must-gather-7tjnr" event={"ID":"dfec70b5-62df-4f31-99d7-e03ace02974a","Type":"ContainerStarted","Data":"d1b52f6c3497955cff7be27a8fd0e5133983b3e4aeb575c0f249d9542aca9d18"}
Sep 17 11:34:01.002367 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:01.002331    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tmn9t/must-gather-7tjnr" event={"ID":"dfec70b5-62df-4f31-99d7-e03ace02974a","Type":"ContainerStarted","Data":"f161690f7cdd22d07109bcb5092fdc60141b26113954fea8b3cc8e17ebe30ab3"}
Sep 17 11:34:01.002846 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:01.002388    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tmn9t/must-gather-7tjnr" event={"ID":"dfec70b5-62df-4f31-99d7-e03ace02974a","Type":"ContainerStarted","Data":"d85de1a14257808f2be7573a35439f85eb530677281ab3122d1fd1f9412f0992"}
Sep 17 11:34:01.023657 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:01.023604    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tmn9t/must-gather-7tjnr" podStartSLOduration=2.032235419 podStartE2EDuration="3.023587889s" totalImagesPullingTime="991.35247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 11:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:34:01.02335517 +0000 UTC m=+2435.121400736" watchObservedRunningTime="2026-09-17 11:34:01.023587889 +0000 UTC m=+2435.121633456"
Sep 17 11:34:02.100404 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:02.100358    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-csvxz_df19ed65-1e04-4637-b6d6-8879cd6b750c/global-pull-secret-syncer/0.log"
Sep 17 11:34:02.274378 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:02.274342    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8gz59_c4efd824-b4e0-4343-9085-553f8e965648/konnectivity-agent/0.log"
Sep 17 11:34:02.431966 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:02.431887    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-204.ec2.internal_d219867a7ad5b4e9ab4856e1757c643d/haproxy/0.log"
Sep 17 11:34:07.695419 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:07.695370    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84cd4f5699-jgql7_8d3bf480-7b79-4675-a30b-25b87d770a43/authorino/0.log"
Sep 17 11:34:07.934513 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:07.934483    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-n598x_3a3cc953-5ff2-42ca-a53f-ae05c57581e3/kuadrant-console-plugin/0.log"
Sep 17 11:34:08.097463 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:08.097434    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-dr8cp_b90e80c8-9401-4712-993c-f72cf2d92c5f/limitador/0.log"
Sep 17 11:34:09.787279 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:09.787252    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f89b5643-d687-4c4f-af65-5bdb43a25f5b/alertmanager/0.log"
Sep 17 11:34:09.826793 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:09.826724    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f89b5643-d687-4c4f-af65-5bdb43a25f5b/config-reloader/0.log"
Sep 17 11:34:09.868631 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:09.868598    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f89b5643-d687-4c4f-af65-5bdb43a25f5b/kube-rbac-proxy-web/0.log"
Sep 17 11:34:09.909947 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:09.909912    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f89b5643-d687-4c4f-af65-5bdb43a25f5b/kube-rbac-proxy/0.log"
Sep 17 11:34:09.964177 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:09.964145    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f89b5643-d687-4c4f-af65-5bdb43a25f5b/kube-rbac-proxy-metric/0.log"
Sep 17 11:34:10.024540 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:10.024505    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f89b5643-d687-4c4f-af65-5bdb43a25f5b/prom-label-proxy/0.log"
Sep 17 11:34:10.072246 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:10.072173    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f89b5643-d687-4c4f-af65-5bdb43a25f5b/init-config-reloader/0.log"
Sep 17 11:34:10.150391 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:10.150358    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-2v8pp_f7c60693-7739-4de7-9a76-13ce9a464eb8/cluster-monitoring-operator/0.log"
Sep 17 11:34:10.361837 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:10.361719    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-dht2n_bda08aa0-553a-4c83-9f6f-fd182889e48f/monitoring-plugin/0.log"
Sep 17 11:34:10.580338 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:10.580307    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t67nf_bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c/node-exporter/0.log"
Sep 17 11:34:10.608513 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:10.608478    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t67nf_bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c/kube-rbac-proxy/0.log"
Sep 17 11:34:10.644081 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:10.644000    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t67nf_bbc5bbc0-4413-4993-b1e8-b5f7fdd5cf4c/init-textfile/0.log"
Sep 17 11:34:11.053895 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:11.053862    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-ppd5x_8d497787-3e87-4725-9e5e-8bde877d03f2/prometheus-operator-admission-webhook/0.log"
Sep 17 11:34:11.090002 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:11.089972    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58c97c79fd-zzkqt_98d9144c-4704-42e8-b203-dcf14e23974a/telemeter-client/0.log"
Sep 17 11:34:11.119002 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:11.118969    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58c97c79fd-zzkqt_98d9144c-4704-42e8-b203-dcf14e23974a/reload/0.log"
Sep 17 11:34:11.151041 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:11.150940    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58c97c79fd-zzkqt_98d9144c-4704-42e8-b203-dcf14e23974a/kube-rbac-proxy/0.log"
Sep 17 11:34:11.185705 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:11.185669    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fc69f886-q7tdw_126afa14-cfb2-47cc-b856-23679e74bf47/thanos-query/0.log"
Sep 17 11:34:11.208462 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:11.208424    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fc69f886-q7tdw_126afa14-cfb2-47cc-b856-23679e74bf47/kube-rbac-proxy-web/0.log"
Sep 17 11:34:11.234110 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:11.234083    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fc69f886-q7tdw_126afa14-cfb2-47cc-b856-23679e74bf47/kube-rbac-proxy/0.log"
Sep 17 11:34:11.289661 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:11.289633    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fc69f886-q7tdw_126afa14-cfb2-47cc-b856-23679e74bf47/prom-label-proxy/0.log"
Sep 17 11:34:11.313981 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:11.313942    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fc69f886-q7tdw_126afa14-cfb2-47cc-b856-23679e74bf47/kube-rbac-proxy-rules/0.log"
Sep 17 11:34:11.335512 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:11.335480    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fc69f886-q7tdw_126afa14-cfb2-47cc-b856-23679e74bf47/kube-rbac-proxy-metrics/0.log"
Sep 17 11:34:11.949137 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:11.949099    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"]
Sep 17 11:34:11.955042 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:11.955011    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:11.964425 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:11.963966    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"]
Sep 17 11:34:12.052782 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:12.052716    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e-lib-modules\") pod \"perf-node-gather-daemonset-glk8p\" (UID: \"4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e\") " pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:12.053142 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:12.053119    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e-sys\") pod \"perf-node-gather-daemonset-glk8p\" (UID: \"4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e\") " pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:12.053357 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:12.053323    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e-proc\") pod \"perf-node-gather-daemonset-glk8p\" (UID: \"4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e\") " pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:12.053524 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:12.053503    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrgvr\" (UniqueName: \"kubernetes.io/projected/4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e-kube-api-access-wrgvr\") pod \"perf-node-gather-daemonset-glk8p\" (UID: \"4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e\") " pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:12.053698 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:12.053685    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e-podres\") pod \"perf-node-gather-daemonset-glk8p\" (UID: \"4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e\") " pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:12.154712 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:12.154674    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e-lib-modules\") pod \"perf-node-gather-daemonset-glk8p\" (UID: \"4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e\") " pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:12.155339 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:12.155316    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e-sys\") pod \"perf-node-gather-daemonset-glk8p\" (UID: \"4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e\") " pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:12.155470 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:12.155001    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e-lib-modules\") pod \"perf-node-gather-daemonset-glk8p\" (UID: \"4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e\") " pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:12.155554 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:12.155365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e-sys\") pod \"perf-node-gather-daemonset-glk8p\" (UID: \"4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e\") " pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:12.155672 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:12.155659    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e-proc\") pod \"perf-node-gather-daemonset-glk8p\" (UID: \"4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e\") " pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:12.155880 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:12.155860    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrgvr\" (UniqueName: \"kubernetes.io/projected/4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e-kube-api-access-wrgvr\") pod \"perf-node-gather-daemonset-glk8p\" (UID: \"4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e\") " pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:12.156368 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:12.156347    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e-podres\") pod \"perf-node-gather-daemonset-glk8p\" (UID: \"4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e\") " pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:12.156635 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:12.156618    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e-podres\") pod \"perf-node-gather-daemonset-glk8p\" (UID: \"4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e\") " pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:12.156747 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:12.155784    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e-proc\") pod \"perf-node-gather-daemonset-glk8p\" (UID: \"4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e\") " pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:12.166837 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:12.166773    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrgvr\" (UniqueName: \"kubernetes.io/projected/4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e-kube-api-access-wrgvr\") pod \"perf-node-gather-daemonset-glk8p\" (UID: \"4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e\") " pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:12.269807 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:12.269709    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:12.563042 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:12.563000    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"]
Sep 17 11:34:12.564902 ip-10-0-139-204 kubenswrapper[2562]: W0917 11:34:12.564870    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4d61d89f_6c43_4f90_87b0_6a3e9d7eb20e.slice/crio-f2432cf9d6396d1c484208ae89918b070204c95b6eb33bba794b3425d4cffbe6 WatchSource:0}: Error finding container f2432cf9d6396d1c484208ae89918b070204c95b6eb33bba794b3425d4cffbe6: Status 404 returned error can't find the container with id f2432cf9d6396d1c484208ae89918b070204c95b6eb33bba794b3425d4cffbe6
Sep 17 11:34:13.090879 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:13.090835    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p" event={"ID":"4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e","Type":"ContainerStarted","Data":"0c9bfd9492e03e9f5d876075d1053e631f866f4710b43cb46591782c8653ef86"}
Sep 17 11:34:13.091089 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:13.090910    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p" event={"ID":"4d61d89f-6c43-4f90-87b0-6a3e9d7eb20e","Type":"ContainerStarted","Data":"f2432cf9d6396d1c484208ae89918b070204c95b6eb33bba794b3425d4cffbe6"}
Sep 17 11:34:13.091804 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:13.091770    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:13.123832 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:13.123756    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p" podStartSLOduration=2.123732253 podStartE2EDuration="2.123732253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 11:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 11:34:13.123037833 +0000 UTC m=+2447.221083400" watchObservedRunningTime="2026-09-17 11:34:13.123732253 +0000 UTC m=+2447.221777819"
Sep 17 11:34:13.275875 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:13.275843    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/1.log"
Sep 17 11:34:13.281387 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:13.281358    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dz7s6_e6482862-22db-436b-97ae-867b9172e290/console-operator/2.log"
Sep 17 11:34:13.895951 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:13.895925    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-54cc58f9b9-28cjv_8f5fe193-073f-4040-9149-cccb6f46a18e/console/0.log"
Sep 17 11:34:13.935909 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:13.935879    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-vd5xp_91f0be6e-7a30-4151-a73a-9c0af5212dc9/download-server/0.log"
Sep 17 11:34:14.678013 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:14.677980    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-frdv7_c5717a5d-758e-4255-a058-84bfefce6fbc/volume-data-source-validator/0.log"
Sep 17 11:34:16.148955 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:16.148927    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jcmwx_bee64b2f-19f8-475f-8f4e-6f0017d52542/dns/0.log"
Sep 17 11:34:16.191654 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:16.191619    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jcmwx_bee64b2f-19f8-475f-8f4e-6f0017d52542/kube-rbac-proxy/0.log"
Sep 17 11:34:16.389791 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:16.389763    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q5vb8_16d6daf4-f47b-4282-8b60-e752eb65e772/dns-node-resolver/0.log"
Sep 17 11:34:17.233778 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:17.233720    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hlqjq_7e7c0e7e-ab26-4238-b048-869055b359b2/node-ca/0.log"
Sep 17 11:34:19.229246 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:19.229220    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8665f874d8-p5sj7_1ad2f5ff-7402-4b66-aae0-606b585eb376/kube-auth-proxy/0.log"
Sep 17 11:34:19.333742 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:19.333703    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7fc56fb846-t46f7_3738db4a-c2eb-42b9-ad15-280d36c6b75e/router/0.log"
Sep 17 11:34:20.115062 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:20.115035    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tmn9t/perf-node-gather-daemonset-glk8p"
Sep 17 11:34:20.239774 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:20.239746    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8tngn_d7e8c9eb-73e6-41a6-992a-a99f6d699061/serve-healthcheck-canary/0.log"
Sep 17 11:34:20.770431 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:20.770390    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-dpdf4_3a72b5be-ac50-4f83-89ab-f0664cf51f1f/insights-operator/0.log"
Sep 17 11:34:20.771363 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:20.771057    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-dpdf4_3a72b5be-ac50-4f83-89ab-f0664cf51f1f/insights-operator/1.log"
Sep 17 11:34:20.798001 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:20.797971    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6vv6s_88983730-d9d0-4e06-9224-f6920bd3f144/kube-rbac-proxy/0.log"
Sep 17 11:34:20.830125 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:20.830096    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6vv6s_88983730-d9d0-4e06-9224-f6920bd3f144/exporter/0.log"
Sep 17 11:34:20.858531 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:20.858503    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6vv6s_88983730-d9d0-4e06-9224-f6920bd3f144/extractor/0.log"
Sep 17 11:34:23.394353 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:23.394319    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8665f874d8-p5sj7_1ad2f5ff-7402-4b66-aae0-606b585eb376/kube-auth-proxy/0.log"
Sep 17 11:34:23.490792 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:23.490698    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7fc56fb846-t46f7_3738db4a-c2eb-42b9-ad15-280d36c6b75e/router/0.log"
Sep 17 11:34:24.155116 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:24.155076    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c54c4bd97-sjrlr_9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e/manager/0.log"
Sep 17 11:34:24.177899 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:24.177867    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c54c4bd97-sjrlr_9dc68714-6c9b-4732-8c7c-9fa5fb1ae69e/copy-manifests/0.log"
Sep 17 11:34:25.449472 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:25.449441    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7bb4f45b46-2gcl9_908cfe1e-afa6-44f7-bae2-c8fe5f8ba786/manager/0.log"
Sep 17 11:34:25.712045 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:25.711975    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-fkhcc_aad44ce7-e30e-48e2-aa16-6e93398ea89f/openshift-lws-operator/0.log"
Sep 17 11:34:25.721261 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:25.721239    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8665f874d8-p5sj7_1ad2f5ff-7402-4b66-aae0-606b585eb376/kube-auth-proxy/0.log"
Sep 17 11:34:25.829834 ip-10-0-139-204 kubenswrapper[2562]: I0917 11:34:25.829784    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7fc56fb846-t46f7_3738db4a-c2eb-42b9-ad15-280d36c6b75e/router/0.log"