Sep 09 03:11:12.326783 ip-10-0-133-131 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 03:11:12.326793 ip-10-0-133-131 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 03:11:12.326800 ip-10-0-133-131 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 03:11:12.327070 ip-10-0-133-131 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 03:11:21.846673 ip-10-0-133-131 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 03:11:21.846692 ip-10-0-133-131 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 2e698eadf8404735a78466833dac2438 --
Sep 09 03:13:02.733915 ip-10-0-133-131 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 03:13:03.187687 ip-10-0-133-131 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 03:13:03.187687 ip-10-0-133-131 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 03:13:03.187687 ip-10-0-133-131 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 03:13:03.187687 ip-10-0-133-131 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 03:13:03.187687 ip-10-0-133-131 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 03:13:03.190265 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.190169    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 03:13:03.194540 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194517    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:13:03.194540 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194535    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:13:03.194540 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194540    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:13:03.194540 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194546    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194551    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194555    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194559    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194562    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194566    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194570    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194574    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194583    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194588    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194609    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194614    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194618    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194622    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194626    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194629    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194633    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194637    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194641    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194645    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:13:03.194801 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194649    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194654    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194658    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194662    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194666    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194670    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194674    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194678    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194682    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194686    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194693    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194699    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194703    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194710    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194716    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194720    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194724    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194729    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194734    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194738    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:13:03.195619 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194743    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194748    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194752    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194756    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194760    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194764    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194769    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194773    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194778    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194784    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194788    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194792    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194797    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194801    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194805    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194809    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194813    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194820    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194826    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:13:03.196490 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194830    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194835    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194840    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194844    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194848    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194852    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194856    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194861    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194866    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194871    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194876    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194880    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194884    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194889    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194893    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194897    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194902    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194905    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194909    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194914    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:13:03.197285 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194918    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194922    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194926    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194931    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194935    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194938    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194943    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194947    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194951    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194955    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194959    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194964    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194968    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194975    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194979    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194983    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.194988    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196178    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196188    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196195    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:13:03.198153 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196199    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196204    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196209    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196214    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196218    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196223    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196228    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196235    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196241    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196246    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196250    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196255    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196260    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196264    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196269    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196274    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196277    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196282    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196286    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:13:03.199051 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196290    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196294    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196298    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196302    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196307    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196311    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196315    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196319    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196324    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196328    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196332    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196337    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196341    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196346    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196350    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196354    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196359    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196363    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196368    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:13:03.199709 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196372    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196377    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196381    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196387    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196392    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196397    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196401    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196405    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196409    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196413    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196419    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196424    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196428    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196434    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196438    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196442    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196446    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196450    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196454    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:13:03.200275 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196458    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196463    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196467    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196471    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196475    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196480    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196484    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196489    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196493    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196497    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196502    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196507    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196511    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196515    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196520    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196524    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196529    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196533    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196537    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196541    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:13:03.200856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196547    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196551    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196556    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196560    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196564    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196568    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196573    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196577    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196581    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196586    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196606    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196610    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196615    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196619    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196624    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196628    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196632    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196637    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.196641    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197270    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:13:03.201389 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197277    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197281    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197285    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197291    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197295    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197299    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197306    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197312    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197317    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197322    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197326    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197331    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197335    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197339    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197344    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197348    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197352    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197356    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197360    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:13:03.201922 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197365    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197370    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197374    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197378    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197382    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197386    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197391    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197394    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197398    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197403    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197407    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197411    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197415    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197419    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197423    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197428    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197432    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197437    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197441    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:13:03.202397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197447    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197451    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197455    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197459    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197464    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197468    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197472    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197476    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197480    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197484    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197488    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197492    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197496    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197500    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197505    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197509    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197513    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197517    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197521    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197525    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:13:03.202885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197529    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197533    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197537    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197541    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197546    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197550    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197554    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197558    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197564    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197568    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197573    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197577    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197581    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197586    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197609    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197614    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197618    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197622    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197626    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197630    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:13:03.203383 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197634    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197638    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197642    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197646    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197650    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197654    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197658    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197663    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197667    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197672    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197676    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197679    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197683    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197687    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197694    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197700    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197705    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197709    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197714    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.197718    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:13:03.203885 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197832    2560 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197893    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197904    2560 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197911    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197919    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197925    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197931    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197938    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197943    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197948    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197954    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197960    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197965    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197970    2560 flags.go:64] FLAG: --cgroup-root=""
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197975    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197980    2560 flags.go:64] FLAG: --client-ca-file=""
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197985    2560 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197989    2560 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.197996    2560 flags.go:64] FLAG: --cluster-domain=""
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198001    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198006    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198011    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198016    2560 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198022    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 03:13:03.204662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198027    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198032    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198037    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198042    2560 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198047    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198051    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198056    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198061    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198067    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198073    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198078    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198083    2560 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198088    2560 flags.go:64] FLAG: --enable-server="true"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198093    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198104    2560 flags.go:64] FLAG: --event-burst="100"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198109    2560 flags.go:64] FLAG: --event-qps="50"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198114    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198119    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198125    2560 flags.go:64] FLAG: --eviction-hard=""
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198130    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198135    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198140    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198145    2560 flags.go:64] FLAG: --eviction-soft=""
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198149    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198154    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 03:13:03.205247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198158    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198162    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198167    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198175    2560 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198180    2560 flags.go:64] FLAG: --feature-gates=""
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198186    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198191    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198196    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198202    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198207    2560 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198211    2560 flags.go:64] FLAG: --help="false"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198216    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-133-131.ec2.internal"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198221    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198226    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198231    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198237    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198244    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198249    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198254    2560 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198259    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198263    2560 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198269    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198275    2560 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198280    2560 flags.go:64] FLAG: --kube-reserved=""
Sep 09 03:13:03.205979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198285    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198290    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198294    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198299    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198304    2560 flags.go:64] FLAG: --lock-file=""
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198308    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198314    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198319    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198328    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198333    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198337    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198344    2560 flags.go:64] FLAG: --logging-format="text"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198349    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198355    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198359    2560 flags.go:64] FLAG: --manifest-url=""
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198364    2560 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198371    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198376    2560 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198382    2560 flags.go:64] FLAG: --max-pods="110"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198387    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198393    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198398    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198402    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198407    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198415    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 03:13:03.206587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198419    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198431    2560 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198436    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198441    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198446    2560 flags.go:64] FLAG: --pod-cidr=""
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198452    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198460    2560 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198465    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198470    2560 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198475    2560 flags.go:64] FLAG: --port="10250"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198480    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198485    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-089db2cc666c13086"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198491    2560 flags.go:64] FLAG: --qos-reserved=""
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198495    2560 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198500    2560 flags.go:64] FLAG: --register-node="true"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198505    2560 flags.go:64] FLAG: --register-with-taints=""
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198511    2560 flags.go:64] FLAG: --registry-burst="10"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198516    2560 flags.go:64] FLAG: --registry-qps="5"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198521    2560 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198527    2560 flags.go:64] FLAG: --reserved-memory=""
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198533    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198538    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198543    2560 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198547    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198552    2560 flags.go:64] FLAG: --runonce="false"
Sep 09 03:13:03.207230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198557    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198562    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198568    2560 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198573    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198577    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198582    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198607    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198613    2560 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198618    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198623    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198628    2560 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198632    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198637    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198643    2560 flags.go:64] FLAG: --system-cgroups=""
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198648    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198656    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198661    2560 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198666    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198672    2560 flags.go:64] FLAG: --tls-min-version=""
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198677    2560 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198681    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198686    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198691    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198696    2560 flags.go:64] FLAG: --v="2"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198702    2560 flags.go:64] FLAG: --version="false"
Sep 09 03:13:03.207910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198708    2560 flags.go:64] FLAG: --vmodule=""
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198716    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.198722    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.198922    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.198929    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.198937    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.198943    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.198949    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.198954    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.198958    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.198962    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.198966    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.198970    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.198978    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.198982    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.198987    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.198991    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.198995    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199000    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199004    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:13:03.208582 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199008    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199013    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199017    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199022    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199026    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199030    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199034    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199039    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199043    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199047    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199050    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199054    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199058    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199064    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199068    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199072    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199076    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199081    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199085    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:13:03.209185 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199090    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199094    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199098    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199103    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199107    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199113    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199124    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199128    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199133    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199137    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199141    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199145    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199149    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199153    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199158    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199163    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199167    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199171    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199175    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:13:03.209684 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199179    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199183    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199187    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199191    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199195    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199199    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199203    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199210    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199214    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199218    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199222    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199226    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199230    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199234    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199241    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199247    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199251    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199256    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199261    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199267    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:13:03.210164 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199272    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199276    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199280    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199284    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199288    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199292    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199296    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199300    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199305    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199309    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199313    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199318    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199321    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199326    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199329    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199334    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199337    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199342    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199346    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199352    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199357    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:13:03.210759 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199360    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:13:03.211283 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199365    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:13:03.211283 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.199369    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:13:03.211283 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.200287    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 03:13:03.211283 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.210769    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 03:13:03.211283 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.210786    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 03:13:03.211283 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.210804    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 03:13:03.211283 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.210810    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 03:13:03.211283 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210877    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:13:03.211283 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210882    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:13:03.211283 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210886    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:13:03.211283 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210888    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:13:03.211283 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210891    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:13:03.211283 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210895    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:13:03.211283 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210897    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:13:03.211283 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210900    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210902    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210905    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210908    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210912    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210917    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210920    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210923    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210926    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210929    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210932    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210934    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210937    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210940    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210942    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210949    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210952    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210954    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210957    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:13:03.211694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210961    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210964    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210967    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210971    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210974    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210977    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210979    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210982    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210984    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.210997    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211001    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211004    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211007    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211010    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211013    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211016    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211019    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211021    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211024    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211034    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:13:03.212211 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211037    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211039    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211042    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211045    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211049    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211053    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211056    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211059    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211062    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211064    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211067    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211069    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211072    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211075    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211077    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211081    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211084    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211087    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211089    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211092    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:13:03.212721 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211094    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211097    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211100    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211103    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211105    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211108    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211110    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211113    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211116    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211118    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211121    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211123    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211134    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211137    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211139    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211142    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211144    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211147    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211150    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211152    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:13:03.213221 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211154    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:13:03.213720 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211162    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:13:03.213720 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211165    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:13:03.213720 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211168    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:13:03.213720 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211170    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:13:03.213720 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211172    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:13:03.213720 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211175    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:13:03.213720 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211178    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:13:03.213720 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211181    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:13:03.213720 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211184    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:13:03.213720 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211186    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:13:03.213720 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211189    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:13:03.213720 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211192    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:13:03.213720 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.211197    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 03:13:03.213720 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211356    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211361    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211365    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211367    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211370    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211373    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211376    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211378    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211381    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211383    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211386    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211394    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211397    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211399    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211402    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211404    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211407    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211409    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211412    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211414    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:13:03.214069 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211417    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211419    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211422    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211424    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211427    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211430    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211432    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211435    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211438    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211440    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211443    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211445    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211448    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211450    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211453    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211455    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211458    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211460    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211463    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:13:03.214554 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211465    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211468    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211470    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211473    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211475    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211483    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211486    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211490    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211493    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211496    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211499    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211501    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211504    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211507    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211510    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211512    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211515    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211517    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211520    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211523    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:13:03.215083 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211526    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211528    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211530    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211533    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211535    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211538    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211540    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211543    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211545    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211548    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211551    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211555    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211557    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211560    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211562    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211565    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211567    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211570    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211578    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:13:03.215568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211580    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211583    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211587    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211604    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211608    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211611    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211614    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211617    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211619    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211622    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211625    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211628    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211631    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211633    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211636    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211639    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211641    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211644    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211646    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211649    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:13:03.216067 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:03.211652    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:13:03.216561 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.211657    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 03:13:03.216561 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.211809    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 03:13:03.216561 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.214369    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 03:13:03.216561 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.215501    2560 server.go:1019] "Starting client certificate rotation"
Sep 09 03:13:03.216561 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.215613    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 03:13:03.216561 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.215652    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 03:13:03.220652 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.220589    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 03:13:03.222885 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.222864    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 03:13:03.232427 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.232405    2560 log.go:25] "Validated CRI v1 runtime API"
Sep 09 03:13:03.236012 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.235995    2560 log.go:25] "Validated CRI v1 image API"
Sep 09 03:13:03.236862 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.236847    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 03:13:03.242674 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.242650    2560 fs.go:135] Filesystem UUIDs: map[35d98704-12d4-4cb9-9c82-3b729dad74d1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fe6377dd-6776-46e4-9d98-37e2e4912f8c:/dev/nvme0n1p3]
Sep 09 03:13:03.242771 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.242672    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 03:13:03.248102 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.247970    2560 manager.go:216] Machine: {Timestamp:2026-09-09 03:13:03.246490778 +0000 UTC m=+0.383238635 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099884 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23c1b574d50809c7ee23196e82d672 SystemUUID:ec23c1b5-74d5-0809-c7ee-23196e82d672 BootID:2e698ead-f840-4735-a784-66833dac2438 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:92:19:a8:14:ef Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:92:19:a8:14:ef Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:2f:ba:6b:c8:06 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 03:13:03.248102 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.248089    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 03:13:03.248254 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.248213    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 03:13:03.250300 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.250271    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 03:13:03.250457 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.250302    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-131.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 03:13:03.251184 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.251171    2560 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 03:13:03.251238 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.251188    2560 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 03:13:03.251238 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.251199    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 03:13:03.251238 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.251218    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 03:13:03.251361 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.251188    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 03:13:03.253637 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.253624    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 03:13:03.258427 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.258413    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 03:13:03.258571 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.258561    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 03:13:03.258624 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.258615    2560 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 03:13:03.258656 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.258640    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 03:13:03.258746 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.258662    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 03:13:03.258746 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.258675    2560 kubelet.go:398] "Adding apiserver pod source"
Sep 09 03:13:03.258746 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.258687    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 03:13:03.259140 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.259125    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 03:13:03.259140 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.259149    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 03:13:03.261672 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.261650    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 03:13:03.265185 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.265168    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 03:13:03.265253 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.265210    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 03:13:03.265469 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.265457    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 03:13:03.265507 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.265478    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 03:13:03.265507 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.265487    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 03:13:03.265507 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.265494    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 03:13:03.265507 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.265504    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 03:13:03.265636 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.265513    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 03:13:03.266926 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.266892    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 03:13:03.266994 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.266930    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 03:13:03.266994 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.266972    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 03:13:03.266994 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.266985    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 03:13:03.267085 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.266997    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 03:13:03.267085 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.267006    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 03:13:03.268863 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.268831    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 03:13:03.268949 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.268899    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 03:13:03.270372 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.270347    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-131.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 03:13:03.270450 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.270368    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 03:13:03.271797 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.271777    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-131.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 03:13:03.273299 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.273283    2560 server.go:1268] "Started kubelet"
Sep 09 03:13:03.273385 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.273341    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 03:13:03.273641 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.273574    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 03:13:03.273748 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.273661    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 03:13:03.274251 ip-10-0-133-131 systemd[1]: Started Kubernetes Kubelet.
Sep 09 03:13:03.275566 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.275552    2560 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 03:13:03.275650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.275630    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 03:13:03.278797 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.278777    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 03:13:03.280214 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.279047    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-131.ec2.internal.18d38885575239a9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-131.ec2.internal,UID:ip-10-0-133-131.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-131.ec2.internal,},FirstTimestamp:2026-09-09 03:13:03.273253289 +0000 UTC m=+0.410001145,LastTimestamp:2026-09-09 03:13:03.273253289 +0000 UTC m=+0.410001145,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-131.ec2.internal,}"
Sep 09 03:13:03.280893 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.280879    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 03:13:03.280958 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.280899    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 03:13:03.280958 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.280948    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Sep 09 03:13:03.281665 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.281638    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 03:13:03.281748 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.281725    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 03:13:03.281748 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.281744    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 03:13:03.281857 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.281801    2560 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 03:13:03.281857 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.281811    2560 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 03:13:03.283608 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.283581    2560 factory.go:55] Registering systemd factory
Sep 09 03:13:03.283702 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.283611    2560 factory.go:223] Registration of the systemd container factory successfully
Sep 09 03:13:03.284084 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.283872    2560 factory.go:153] Registering CRI-O factory
Sep 09 03:13:03.284084 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.283885    2560 factory.go:223] Registration of the crio container factory successfully
Sep 09 03:13:03.284084 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.283957    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 03:13:03.284084 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.283975    2560 factory.go:103] Registering Raw factory
Sep 09 03:13:03.284084 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.283988    2560 manager.go:1196] Started watching for new ooms in manager
Sep 09 03:13:03.284842 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.284410    2560 manager.go:349] Starting recovery of all containers
Sep 09 03:13:03.286306 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.286263    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-131.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 03:13:03.286583 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.286457    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 03:13:03.299560 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.299540    2560 manager.go:354] Recovery completed
Sep 09 03:13:03.299926 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.299897    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-djjgr"
Sep 09 03:13:03.305202 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.305185    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:13:03.307223 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.307204    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:13:03.307303 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.307240    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:13:03.307303 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.307255    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:13:03.307918 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.307899    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 03:13:03.307918 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.307909    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:13:03.308049 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.307910    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 03:13:03.308049 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.307977    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 03:13:03.308786 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.308753    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:13:03.308853 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.308787    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:13:03.308853 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.308801    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:13:03.308853 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.308831    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-djjgr"
Sep 09 03:13:03.309301 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.309236    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-131.ec2.internal.18d3888559589156  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-131.ec2.internal,UID:ip-10-0-133-131.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-131.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-131.ec2.internal,},FirstTimestamp:2026-09-09 03:13:03.307223382 +0000 UTC m=+0.443971249,LastTimestamp:2026-09-09 03:13:03.307223382 +0000 UTC m=+0.443971249,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-131.ec2.internal,}"
Sep 09 03:13:03.310204 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.310190    2560 policy_none.go:49] "None policy: Start"
Sep 09 03:13:03.310259 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.310208    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 03:13:03.310259 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.310222    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 03:13:03.312163 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.312152    2560 policy_none.go:47] "Start"
Sep 09 03:13:03.360159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.346292    2560 manager.go:337] "Starting Device Plugin manager"
Sep 09 03:13:03.360159 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.346331    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 03:13:03.360159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.346344    2560 server.go:87] "Starting device plugin registration server"
Sep 09 03:13:03.360159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.346698    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 03:13:03.360159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.346710    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 03:13:03.360159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.346813    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 03:13:03.360159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.346897    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 03:13:03.360159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.346908    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 03:13:03.360159 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.347762    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 03:13:03.360159 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.347813    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-131.ec2.internal\" not found"
Sep 09 03:13:03.360159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.350695    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 03:13:03.360159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.351956    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 03:13:03.360159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.351996    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 03:13:03.360159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.352018    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 03:13:03.360159 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.352058    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 03:13:03.360159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.353615    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:13:03.447464 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.447225    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:13:03.448432 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.448415    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:13:03.448497 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.448445    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:13:03.448497 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.448456    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:13:03.448497 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.448480    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-131.ec2.internal"
Sep 09 03:13:03.452289 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.452272    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"]
Sep 09 03:13:03.452357 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.452348    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:13:03.453815 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.453801    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:13:03.453886 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.453825    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:13:03.453886 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.453835    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:13:03.456199 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.456185    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:13:03.456338 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.456323    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
Sep 09 03:13:03.456395 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.456359    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:13:03.456884 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.456869    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:13:03.456936 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.456898    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:13:03.456968 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.456873    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:13:03.456998 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.456979    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:13:03.456998 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.456991    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:13:03.457088 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.456951    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:13:03.457186 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.457083    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-131.ec2.internal"
Sep 09 03:13:03.457186 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.457108    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-131.ec2.internal\": node \"ip-10-0-133-131.ec2.internal\" not found"
Sep 09 03:13:03.459323 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.459309    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"
Sep 09 03:13:03.459377 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.459333    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:13:03.460022 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.460007    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:13:03.460080 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.460033    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:13:03.460080 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.460043    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:13:03.477152 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.477133    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-131.ec2.internal\" not found" node="ip-10-0-133-131.ec2.internal"
Sep 09 03:13:03.480394 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.480373    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-131.ec2.internal\" not found" node="ip-10-0-133-131.ec2.internal"
Sep 09 03:13:03.483042 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.483025    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Sep 09 03:13:03.582937 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.582904    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c3c6a94ac7c6993af73a9ccfb982bee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal\" (UID: \"2c3c6a94ac7c6993af73a9ccfb982bee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
Sep 09 03:13:03.582937 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.582942    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c3c6a94ac7c6993af73a9ccfb982bee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal\" (UID: \"2c3c6a94ac7c6993af73a9ccfb982bee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
Sep 09 03:13:03.583105 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.582963    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/50ae6cedb6cbe5050a1216b96fc89c4f-config\") pod \"kube-apiserver-proxy-ip-10-0-133-131.ec2.internal\" (UID: \"50ae6cedb6cbe5050a1216b96fc89c4f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"
Sep 09 03:13:03.583951 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.583930    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Sep 09 03:13:03.683341 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.683312    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c3c6a94ac7c6993af73a9ccfb982bee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal\" (UID: \"2c3c6a94ac7c6993af73a9ccfb982bee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
Sep 09 03:13:03.683429 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.683342    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c3c6a94ac7c6993af73a9ccfb982bee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal\" (UID: \"2c3c6a94ac7c6993af73a9ccfb982bee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
Sep 09 03:13:03.683429 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.683371    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/50ae6cedb6cbe5050a1216b96fc89c4f-config\") pod \"kube-apiserver-proxy-ip-10-0-133-131.ec2.internal\" (UID: \"50ae6cedb6cbe5050a1216b96fc89c4f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"
Sep 09 03:13:03.683429 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.683400    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/50ae6cedb6cbe5050a1216b96fc89c4f-config\") pod \"kube-apiserver-proxy-ip-10-0-133-131.ec2.internal\" (UID: \"50ae6cedb6cbe5050a1216b96fc89c4f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"
Sep 09 03:13:03.683429 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.683409    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c3c6a94ac7c6993af73a9ccfb982bee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal\" (UID: \"2c3c6a94ac7c6993af73a9ccfb982bee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
Sep 09 03:13:03.683429 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.683419    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c3c6a94ac7c6993af73a9ccfb982bee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal\" (UID: \"2c3c6a94ac7c6993af73a9ccfb982bee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
Sep 09 03:13:03.684349 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.684334    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Sep 09 03:13:03.780961 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.780893    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
Sep 09 03:13:03.784177 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:03.784157    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"
Sep 09 03:13:03.784688 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.784667    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Sep 09 03:13:03.885684 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.885651    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Sep 09 03:13:03.986639 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:03.986577    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Sep 09 03:13:04.087783 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:04.087691    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Sep 09 03:13:04.181040 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:04.181006    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:13:04.187929 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:04.187900    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Sep 09 03:13:04.215148 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:04.215116    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 03:13:04.215294 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:04.215276    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 03:13:04.215366 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:04.215298    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 03:13:04.288418 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:04.288392    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Sep 09 03:13:04.311933 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:04.311897    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 03:08:04 +0000 UTC" deadline="2028-02-03 07:17:17.614172547 +0000 UTC"
Sep 09 03:13:04.312095 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:04.311933    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12292h4m13.30224775s"
Sep 09 03:13:04.386509 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:04.386437    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:13:04.388627 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:04.388607    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Sep 09 03:13:04.457251 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:04.457218    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3c6a94ac7c6993af73a9ccfb982bee.slice/crio-3dc10ac35d19fc0c031b743d3255c0d47664c63691a6fe3fbe335eb5ff263433 WatchSource:0}: Error finding container 3dc10ac35d19fc0c031b743d3255c0d47664c63691a6fe3fbe335eb5ff263433: Status 404 returned error can't find the container with id 3dc10ac35d19fc0c031b743d3255c0d47664c63691a6fe3fbe335eb5ff263433
Sep 09 03:13:04.465462 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:04.465442    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:13:04.468390 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:04.468355    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ae6cedb6cbe5050a1216b96fc89c4f.slice/crio-b5b775f327c38b29b9fcc9cfa9ae48baa53a5c7c104b45707d63d6be236583a3 WatchSource:0}: Error finding container b5b775f327c38b29b9fcc9cfa9ae48baa53a5c7c104b45707d63d6be236583a3: Status 404 returned error can't find the container with id b5b775f327c38b29b9fcc9cfa9ae48baa53a5c7c104b45707d63d6be236583a3
Sep 09 03:13:04.489545 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:04.489512    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Sep 09 03:13:04.590152 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:04.590124    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
Sep 09 03:13:04.644659 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:04.644559    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:13:04.683583 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:04.683552    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
Sep 09 03:13:04.697505 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:04.697481    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 03:13:04.698789 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:04.698774    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"
Sep 09 03:13:04.705396 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:04.705382    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 03:13:05.259544 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.259245    2560 apiserver.go:52] "Watching apiserver"
Sep 09 03:13:05.268463 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.268432    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 03:13:05.269022 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.268997    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-zxq68","openshift-network-diagnostics/network-check-target-tls5w","openshift-network-operator/iptables-alerter-7nq7n","kube-system/global-pull-secret-syncer-v89sv","kube-system/konnectivity-agent-7xspc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb","openshift-dns/node-resolver-sbmb7","openshift-multus/network-metrics-daemon-hlmbb","openshift-ovn-kubernetes/ovnkube-node-cjmpt","kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal","openshift-cluster-node-tuning-operator/tuned-rlqww","openshift-image-registry/node-ca-v47sj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal","openshift-multus/multus-additional-cni-plugins-sbbhj"]
Sep 09 03:13:05.271743 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.271693    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sbmb7"
Sep 09 03:13:05.274490 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.273990    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.275060 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.274826    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 03:13:05.275060 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.274901    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 03:13:05.275060 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.275041    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hrwmn\""
Sep 09 03:13:05.276810 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.276756    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7nq7n"
Sep 09 03:13:05.276898 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.276819    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:13:05.277227 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.277204    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kdfzf\""
Sep 09 03:13:05.277327 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.277307    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 03:13:05.279244 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.279219    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:05.279343 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:05.279282    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v89sv" podUID="422df607-4fd9-4edd-ad99-7b6214c7d970"
Sep 09 03:13:05.280042 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.279581    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:13:05.280042 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.279825    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 03:13:05.280633 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.280441    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k476f\""
Sep 09 03:13:05.280633 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.280501    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 03:13:05.282183 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.281852    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7xspc"
Sep 09 03:13:05.284935 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.284187    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 03:13:05.284935 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.284842    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 03:13:05.285114 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.285099    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.287409 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.287386    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-b9mlp\""
Sep 09 03:13:05.287512 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.287435    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 03:13:05.288347 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.288123    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tt6wv\""
Sep 09 03:13:05.288347 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.288235    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 03:13:05.288347 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.288123    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 03:13:05.288550 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.288438    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 03:13:05.288550 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.288489    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.291436 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.291169    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.291837 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.291818    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 03:13:05.291940 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.291835    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 03:13:05.292094 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.292074    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 03:13:05.292879 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.292859    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z4h5g\""
Sep 09 03:13:05.292973 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.292909    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 03:13:05.293442 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.293424    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 03:13:05.293494 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.293467    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 03:13:05.293804 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.293783    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 03:13:05.293889 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.293843    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:05.293993    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tls5w" podUID="d539727e-a1d4-43f5-be9e-84c6611912be"
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.294113    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kfxjx\""
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.294326    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.294357    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.294587    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/422df607-4fd9-4edd-ad99-7b6214c7d970-dbus\") pod \"global-pull-secret-syncer-v89sv\" (UID: \"422df607-4fd9-4edd-ad99-7b6214c7d970\") " pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.294617    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e698c58a-191f-485e-8258-fae42a7238e3-device-dir\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.294641    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-run\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.294659    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.294665    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k89b\" (UniqueName: \"kubernetes.io/projected/965c7f4b-4baa-474b-a29e-8eb699e4752e-kube-api-access-2k89b\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.294704    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.294710    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e698c58a-191f-485e-8258-fae42a7238e3-socket-dir\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.294736    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcp94\" (UniqueName: \"kubernetes.io/projected/e698c58a-191f-485e-8258-fae42a7238e3-kube-api-access-hcp94\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.294761    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/965c7f4b-4baa-474b-a29e-8eb699e4752e-tmp\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.294783    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e79f2fe7-aee6-4fe3-9834-1495344fc754-host-slash\") pod \"iptables-alerter-7nq7n\" (UID: \"e79f2fe7-aee6-4fe3-9834-1495344fc754\") " pod="openshift-network-operator/iptables-alerter-7nq7n"
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.294816    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e698c58a-191f-485e-8258-fae42a7238e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.294848    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-sysconfig\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.294877    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-sysctl-conf\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.294924    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-systemd\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.294954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.294952    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-host\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.296205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295024    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82jw\" (UniqueName: \"kubernetes.io/projected/e79f2fe7-aee6-4fe3-9834-1495344fc754-kube-api-access-g82jw\") pod \"iptables-alerter-7nq7n\" (UID: \"e79f2fe7-aee6-4fe3-9834-1495344fc754\") " pod="openshift-network-operator/iptables-alerter-7nq7n"
Sep 09 03:13:05.296205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295064    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/def2fdc5-08ed-43a3-b9c1-b8b55e6444c5-hosts-file\") pod \"node-resolver-sbmb7\" (UID: \"def2fdc5-08ed-43a3-b9c1-b8b55e6444c5\") " pod="openshift-dns/node-resolver-sbmb7"
Sep 09 03:13:05.296205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295125    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-tuned\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.296205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295151    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret\") pod \"global-pull-secret-syncer-v89sv\" (UID: \"422df607-4fd9-4edd-ad99-7b6214c7d970\") " pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:05.296205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295172    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-sys\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.296205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295207    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-var-lib-kubelet\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.296205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e698c58a-191f-485e-8258-fae42a7238e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.296205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295266    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e698c58a-191f-485e-8258-fae42a7238e3-registration-dir\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.296205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-kubernetes\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.296205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295297    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-sysctl-d\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.296205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8465770f-1ae2-4566-8c6d-12cc331da1ec-konnectivity-ca\") pod \"konnectivity-agent-7xspc\" (UID: \"8465770f-1ae2-4566-8c6d-12cc331da1ec\") " pod="kube-system/konnectivity-agent-7xspc"
Sep 09 03:13:05.296205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295352    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch9gv\" (UniqueName: \"kubernetes.io/projected/def2fdc5-08ed-43a3-b9c1-b8b55e6444c5-kube-api-access-ch9gv\") pod \"node-resolver-sbmb7\" (UID: \"def2fdc5-08ed-43a3-b9c1-b8b55e6444c5\") " pod="openshift-dns/node-resolver-sbmb7"
Sep 09 03:13:05.296205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295391    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-modprobe-d\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.296205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295417    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-lib-modules\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.296205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295441    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/422df607-4fd9-4edd-ad99-7b6214c7d970-kubelet-config\") pod \"global-pull-secret-syncer-v89sv\" (UID: \"422df607-4fd9-4edd-ad99-7b6214c7d970\") " pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:05.296205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295461    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8465770f-1ae2-4566-8c6d-12cc331da1ec-agent-certs\") pod \"konnectivity-agent-7xspc\" (UID: \"8465770f-1ae2-4566-8c6d-12cc331da1ec\") " pod="kube-system/konnectivity-agent-7xspc"
Sep 09 03:13:05.297009 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295476    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e698c58a-191f-485e-8258-fae42a7238e3-sys-fs\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.297009 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295496    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def2fdc5-08ed-43a3-b9c1-b8b55e6444c5-tmp-dir\") pod \"node-resolver-sbmb7\" (UID: \"def2fdc5-08ed-43a3-b9c1-b8b55e6444c5\") " pod="openshift-dns/node-resolver-sbmb7"
Sep 09 03:13:05.297009 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.295519    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e79f2fe7-aee6-4fe3-9834-1495344fc754-iptables-alerter-script\") pod \"iptables-alerter-7nq7n\" (UID: \"e79f2fe7-aee6-4fe3-9834-1495344fc754\") " pod="openshift-network-operator/iptables-alerter-7nq7n"
Sep 09 03:13:05.297009 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.296319    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:05.297009 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:05.296378    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:13:05.298073 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.298054    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 03:13:05.299358 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.299339    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.301751 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.301732    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 03:13:05.301964 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.301943    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v47sj"
Sep 09 03:13:05.302479 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.302459    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tz2vp\""
Sep 09 03:13:05.304447 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.304410    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q7lsd\""
Sep 09 03:13:05.304558 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.304534    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 03:13:05.304696 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.304679    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 03:13:05.304784 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.304749    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 03:13:05.313407 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.313384    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qc4gj"
Sep 09 03:13:05.321310 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.321279    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qc4gj"
Sep 09 03:13:05.359716 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.359640    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal" event={"ID":"50ae6cedb6cbe5050a1216b96fc89c4f","Type":"ContainerStarted","Data":"b5b775f327c38b29b9fcc9cfa9ae48baa53a5c7c104b45707d63d6be236583a3"}
Sep 09 03:13:05.361605 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.361577    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal" event={"ID":"2c3c6a94ac7c6993af73a9ccfb982bee","Type":"ContainerStarted","Data":"3dc10ac35d19fc0c031b743d3255c0d47664c63691a6fe3fbe335eb5ff263433"}
Sep 09 03:13:05.383614 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.383591    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 03:13:05.395856 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.395827    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7w7g\" (UniqueName: \"kubernetes.io/projected/7e455be0-bf16-41ad-a378-7621bcf270c8-kube-api-access-n7w7g\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.395981 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.395870    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k89b\" (UniqueName: \"kubernetes.io/projected/965c7f4b-4baa-474b-a29e-8eb699e4752e-kube-api-access-2k89b\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.395981 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.395896    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e698c58a-191f-485e-8258-fae42a7238e3-socket-dir\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.395981 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.395918    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcp94\" (UniqueName: \"kubernetes.io/projected/e698c58a-191f-485e-8258-fae42a7238e3-kube-api-access-hcp94\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.395981 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.395942    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs\") pod \"network-metrics-daemon-hlmbb\" (UID: \"f8508862-2ffb-4bd8-bb4f-67cd69a78075\") " pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:05.395981 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.395970    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e455be0-bf16-41ad-a378-7621bcf270c8-ovnkube-script-lib\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.396246 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.395995    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-multus-socket-dir-parent\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.396246 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396022    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-sysconfig\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.396246 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396046    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-sysctl-conf\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.396246 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396069    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-systemd\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.396246 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396094    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-host\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.396246 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396120    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/def2fdc5-08ed-43a3-b9c1-b8b55e6444c5-hosts-file\") pod \"node-resolver-sbmb7\" (UID: \"def2fdc5-08ed-43a3-b9c1-b8b55e6444c5\") " pod="openshift-dns/node-resolver-sbmb7"
Sep 09 03:13:05.396246 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396144    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-tuned\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.396246 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396170    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret\") pod \"global-pull-secret-syncer-v89sv\" (UID: \"422df607-4fd9-4edd-ad99-7b6214c7d970\") " pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:05.396246 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396197    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b3084e8-6418-4139-9623-5528c6b2cef3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.396246 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396223    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.396766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396251    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e455be0-bf16-41ad-a378-7621bcf270c8-ovn-node-metrics-cert\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.396766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396278    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-var-lib-kubelet\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.396766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396301    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e698c58a-191f-485e-8258-fae42a7238e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.396766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396330    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-host-var-lib-cni-multus\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.396766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396355    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-systemd-units\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.396766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396383    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e455be0-bf16-41ad-a378-7621bcf270c8-ovnkube-config\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.396766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396424    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl5z2\" (UniqueName: \"kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2\") pod \"network-check-target-tls5w\" (UID: \"d539727e-a1d4-43f5-be9e-84c6611912be\") " pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:05.396766 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:05.396446    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:13:05.396766 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:05.396512    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret podName:422df607-4fd9-4edd-ad99-7b6214c7d970 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:05.896495563 +0000 UTC m=+3.033243433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret") pod "global-pull-secret-syncer-v89sv" (UID: "422df607-4fd9-4edd-ad99-7b6214c7d970") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:13:05.396766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396513    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-kubernetes\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.396766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396547    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-var-lib-kubelet\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.396766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396577    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-sysconfig\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.396766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396604    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e698c58a-191f-485e-8258-fae42a7238e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.396766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396449    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-kubernetes\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.396766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396654    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-systemd\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.396766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396712    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-host\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.396766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396714    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-sysctl-d\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.397650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396757    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8465770f-1ae2-4566-8c6d-12cc331da1ec-konnectivity-ca\") pod \"konnectivity-agent-7xspc\" (UID: \"8465770f-1ae2-4566-8c6d-12cc331da1ec\") " pod="kube-system/konnectivity-agent-7xspc"
Sep 09 03:13:05.397650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396785    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-system-cni-dir\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.397650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396811    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/88f6921c-8fc8-45b4-b4ac-368be491377b-multus-daemon-config\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.397650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396837    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f4c4cf1-bf9c-499d-b25e-fe93588e6c58-host\") pod \"node-ca-v47sj\" (UID: \"0f4c4cf1-bf9c-499d-b25e-fe93588e6c58\") " pod="openshift-image-registry/node-ca-v47sj"
Sep 09 03:13:05.397650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396840    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-sysctl-d\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.397650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-run-netns\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.397650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396889    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-modprobe-d\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.397650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396914    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-lib-modules\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.397650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396942    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/422df607-4fd9-4edd-ad99-7b6214c7d970-kubelet-config\") pod \"global-pull-secret-syncer-v89sv\" (UID: \"422df607-4fd9-4edd-ad99-7b6214c7d970\") " pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:05.397650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396968    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8465770f-1ae2-4566-8c6d-12cc331da1ec-agent-certs\") pod \"konnectivity-agent-7xspc\" (UID: \"8465770f-1ae2-4566-8c6d-12cc331da1ec\") " pod="kube-system/konnectivity-agent-7xspc"
Sep 09 03:13:05.397650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397022    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e698c58a-191f-485e-8258-fae42a7238e3-sys-fs\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.397650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397049    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9vc5\" (UniqueName: \"kubernetes.io/projected/88f6921c-8fc8-45b4-b4ac-368be491377b-kube-api-access-b9vc5\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.397650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397079    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-run-openvswitch\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.397650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397103    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-sysctl-conf\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.397650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397107    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def2fdc5-08ed-43a3-b9c1-b8b55e6444c5-tmp-dir\") pod \"node-resolver-sbmb7\" (UID: \"def2fdc5-08ed-43a3-b9c1-b8b55e6444c5\") " pod="openshift-dns/node-resolver-sbmb7"
Sep 09 03:13:05.397650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397154    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e79f2fe7-aee6-4fe3-9834-1495344fc754-iptables-alerter-script\") pod \"iptables-alerter-7nq7n\" (UID: \"e79f2fe7-aee6-4fe3-9834-1495344fc754\") " pod="openshift-network-operator/iptables-alerter-7nq7n"
Sep 09 03:13:05.397650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397168    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-modprobe-d\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.398533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397181    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/422df607-4fd9-4edd-ad99-7b6214c7d970-dbus\") pod \"global-pull-secret-syncer-v89sv\" (UID: \"422df607-4fd9-4edd-ad99-7b6214c7d970\") " pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:05.398533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397209    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e698c58a-191f-485e-8258-fae42a7238e3-device-dir\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.398533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397230    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/def2fdc5-08ed-43a3-b9c1-b8b55e6444c5-hosts-file\") pod \"node-resolver-sbmb7\" (UID: \"def2fdc5-08ed-43a3-b9c1-b8b55e6444c5\") " pod="openshift-dns/node-resolver-sbmb7"
Sep 09 03:13:05.398533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.396991    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e698c58a-191f-485e-8258-fae42a7238e3-socket-dir\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.398533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397338    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-lib-modules\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.398533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397379    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/422df607-4fd9-4edd-ad99-7b6214c7d970-kubelet-config\") pod \"global-pull-secret-syncer-v89sv\" (UID: \"422df607-4fd9-4edd-ad99-7b6214c7d970\") " pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:05.398533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397392    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/422df607-4fd9-4edd-ad99-7b6214c7d970-dbus\") pod \"global-pull-secret-syncer-v89sv\" (UID: \"422df607-4fd9-4edd-ad99-7b6214c7d970\") " pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:05.398533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397424    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvbn6\" (UniqueName: \"kubernetes.io/projected/4b3084e8-6418-4139-9623-5528c6b2cef3-kube-api-access-hvbn6\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.398533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def2fdc5-08ed-43a3-b9c1-b8b55e6444c5-tmp-dir\") pod \"node-resolver-sbmb7\" (UID: \"def2fdc5-08ed-43a3-b9c1-b8b55e6444c5\") " pod="openshift-dns/node-resolver-sbmb7"
Sep 09 03:13:05.398533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397485    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-host-var-lib-kubelet\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.398533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397494    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8465770f-1ae2-4566-8c6d-12cc331da1ec-konnectivity-ca\") pod \"konnectivity-agent-7xspc\" (UID: \"8465770f-1ae2-4566-8c6d-12cc331da1ec\") " pod="kube-system/konnectivity-agent-7xspc"
Sep 09 03:13:05.398533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397533    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-host-run-multus-certs\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.398533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397615    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-run\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.398533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397623    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e698c58a-191f-485e-8258-fae42a7238e3-sys-fs\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.398533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397702    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e698c58a-191f-485e-8258-fae42a7238e3-device-dir\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.398533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397694    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b3084e8-6418-4139-9623-5528c6b2cef3-system-cni-dir\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.398533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397721    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 03:13:05.399330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397746    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b3084e8-6418-4139-9623-5528c6b2cef3-cnibin\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.399330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397748    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-run\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.399330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397788    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-cnibin\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.399330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397817    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-kubelet\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.399330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397841    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-var-lib-openvswitch\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.399330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397871    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/965c7f4b-4baa-474b-a29e-8eb699e4752e-tmp\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.399330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397895    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e79f2fe7-aee6-4fe3-9834-1495344fc754-host-slash\") pod \"iptables-alerter-7nq7n\" (UID: \"e79f2fe7-aee6-4fe3-9834-1495344fc754\") " pod="openshift-network-operator/iptables-alerter-7nq7n"
Sep 09 03:13:05.399330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397921    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e698c58a-191f-485e-8258-fae42a7238e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.399330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397945    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-multus-cni-dir\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.399330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.397978    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-host-run-k8s-cni-cncf-io\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.399330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398017    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f4c4cf1-bf9c-499d-b25e-fe93588e6c58-serviceca\") pod \"node-ca-v47sj\" (UID: \"0f4c4cf1-bf9c-499d-b25e-fe93588e6c58\") " pod="openshift-image-registry/node-ca-v47sj"
Sep 09 03:13:05.399330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398043    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-log-socket\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.399330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398069    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82jw\" (UniqueName: \"kubernetes.io/projected/e79f2fe7-aee6-4fe3-9834-1495344fc754-kube-api-access-g82jw\") pod \"iptables-alerter-7nq7n\" (UID: \"e79f2fe7-aee6-4fe3-9834-1495344fc754\") " pod="openshift-network-operator/iptables-alerter-7nq7n"
Sep 09 03:13:05.399330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398079    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e79f2fe7-aee6-4fe3-9834-1495344fc754-host-slash\") pod \"iptables-alerter-7nq7n\" (UID: \"e79f2fe7-aee6-4fe3-9834-1495344fc754\") " pod="openshift-network-operator/iptables-alerter-7nq7n"
Sep 09 03:13:05.399330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398097    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88f6921c-8fc8-45b4-b4ac-368be491377b-cni-binary-copy\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.399330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398123    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.399330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398147    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-run-ovn\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.400242 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398171    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-sys\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.400242 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e698c58a-191f-485e-8258-fae42a7238e3-registration-dir\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.400242 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398225    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b3084e8-6418-4139-9623-5528c6b2cef3-cni-binary-copy\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.400242 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398248    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-hostroot\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.400242 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398273    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-multus-conf-dir\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.400242 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398298    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74dqp\" (UniqueName: \"kubernetes.io/projected/0f4c4cf1-bf9c-499d-b25e-fe93588e6c58-kube-api-access-74dqp\") pod \"node-ca-v47sj\" (UID: \"0f4c4cf1-bf9c-499d-b25e-fe93588e6c58\") " pod="openshift-image-registry/node-ca-v47sj"
Sep 09 03:13:05.400242 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398336    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-run-systemd\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.400242 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398365    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b3084e8-6418-4139-9623-5528c6b2cef3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.400242 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398390    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-etc-kubernetes\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.400242 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398414    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-slash\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.400242 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398438    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-node-log\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.400242 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398461    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-cni-netd\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.400242 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398474    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e79f2fe7-aee6-4fe3-9834-1495344fc754-iptables-alerter-script\") pod \"iptables-alerter-7nq7n\" (UID: \"e79f2fe7-aee6-4fe3-9834-1495344fc754\") " pod="openshift-network-operator/iptables-alerter-7nq7n"
Sep 09 03:13:05.400242 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398489    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch9gv\" (UniqueName: \"kubernetes.io/projected/def2fdc5-08ed-43a3-b9c1-b8b55e6444c5-kube-api-access-ch9gv\") pod \"node-resolver-sbmb7\" (UID: \"def2fdc5-08ed-43a3-b9c1-b8b55e6444c5\") " pod="openshift-dns/node-resolver-sbmb7"
Sep 09 03:13:05.400242 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398517    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b3084e8-6418-4139-9623-5528c6b2cef3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.400242 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398571    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-host-var-lib-cni-bin\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.400242 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398580    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/965c7f4b-4baa-474b-a29e-8eb699e4752e-sys\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.401122 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398615    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e698c58a-191f-485e-8258-fae42a7238e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.401122 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398688    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e698c58a-191f-485e-8258-fae42a7238e3-registration-dir\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.401122 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398774    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-cni-bin\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.401122 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398802    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e455be0-bf16-41ad-a378-7621bcf270c8-env-overrides\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.401122 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398828    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp57z\" (UniqueName: \"kubernetes.io/projected/f8508862-2ffb-4bd8-bb4f-67cd69a78075-kube-api-access-zp57z\") pod \"network-metrics-daemon-hlmbb\" (UID: \"f8508862-2ffb-4bd8-bb4f-67cd69a78075\") " pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:05.401122 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398854    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-host-run-netns\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.401122 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.398882    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-etc-openvswitch\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.402640 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.402440    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/965c7f4b-4baa-474b-a29e-8eb699e4752e-etc-tuned\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.402640 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.402481    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/965c7f4b-4baa-474b-a29e-8eb699e4752e-tmp\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.402847 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.402641    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8465770f-1ae2-4566-8c6d-12cc331da1ec-agent-certs\") pod \"konnectivity-agent-7xspc\" (UID: \"8465770f-1ae2-4566-8c6d-12cc331da1ec\") " pod="kube-system/konnectivity-agent-7xspc"
Sep 09 03:13:05.404520 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.404465    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcp94\" (UniqueName: \"kubernetes.io/projected/e698c58a-191f-485e-8258-fae42a7238e3-kube-api-access-hcp94\") pod \"aws-ebs-csi-driver-node-l2hqb\" (UID: \"e698c58a-191f-485e-8258-fae42a7238e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.407995 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.407961    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k89b\" (UniqueName: \"kubernetes.io/projected/965c7f4b-4baa-474b-a29e-8eb699e4752e-kube-api-access-2k89b\") pod \"tuned-rlqww\" (UID: \"965c7f4b-4baa-474b-a29e-8eb699e4752e\") " pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.409656 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.409627    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82jw\" (UniqueName: \"kubernetes.io/projected/e79f2fe7-aee6-4fe3-9834-1495344fc754-kube-api-access-g82jw\") pod \"iptables-alerter-7nq7n\" (UID: \"e79f2fe7-aee6-4fe3-9834-1495344fc754\") " pod="openshift-network-operator/iptables-alerter-7nq7n"
Sep 09 03:13:05.410297 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.410271    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch9gv\" (UniqueName: \"kubernetes.io/projected/def2fdc5-08ed-43a3-b9c1-b8b55e6444c5-kube-api-access-ch9gv\") pod \"node-resolver-sbmb7\" (UID: \"def2fdc5-08ed-43a3-b9c1-b8b55e6444c5\") " pod="openshift-dns/node-resolver-sbmb7"
Sep 09 03:13:05.480047 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.479965    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:13:05.500122 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500088    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-host-var-lib-cni-bin\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.500261 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500134    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-cni-bin\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.500261 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500162    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e455be0-bf16-41ad-a378-7621bcf270c8-env-overrides\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.500261 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500181    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp57z\" (UniqueName: \"kubernetes.io/projected/f8508862-2ffb-4bd8-bb4f-67cd69a78075-kube-api-access-zp57z\") pod \"network-metrics-daemon-hlmbb\" (UID: \"f8508862-2ffb-4bd8-bb4f-67cd69a78075\") " pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:05.500261 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-host-run-netns\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.500261 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500215    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-etc-openvswitch\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.500261 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500224    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-cni-bin\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.500261 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500223    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-host-var-lib-cni-bin\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.500644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500232    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7w7g\" (UniqueName: \"kubernetes.io/projected/7e455be0-bf16-41ad-a378-7621bcf270c8-kube-api-access-n7w7g\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.500644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500287    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-etc-openvswitch\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.500644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500307    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs\") pod \"network-metrics-daemon-hlmbb\" (UID: \"f8508862-2ffb-4bd8-bb4f-67cd69a78075\") " pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:05.500644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500350    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e455be0-bf16-41ad-a378-7621bcf270c8-ovnkube-script-lib\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.500644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500380    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-multus-socket-dir-parent\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.500644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500431    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b3084e8-6418-4139-9623-5528c6b2cef3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.500644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500485    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.500644 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:05.500509    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:13:05.500644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500516    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e455be0-bf16-41ad-a378-7621bcf270c8-ovn-node-metrics-cert\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.500644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500557    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-host-var-lib-cni-multus\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.500644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500557    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-host-run-netns\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.500644 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:05.500572    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs podName:f8508862-2ffb-4bd8-bb4f-67cd69a78075 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:06.000553317 +0000 UTC m=+3.137301167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs") pod "network-metrics-daemon-hlmbb" (UID: "f8508862-2ffb-4bd8-bb4f-67cd69a78075") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:13:05.500644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500625    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-systemd-units\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.500644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500621    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-host-var-lib-cni-multus\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.500644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500641    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.501444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500667    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e455be0-bf16-41ad-a378-7621bcf270c8-ovnkube-config\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.501444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500693    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl5z2\" (UniqueName: \"kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2\") pod \"network-check-target-tls5w\" (UID: \"d539727e-a1d4-43f5-be9e-84c6611912be\") " pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:05.501444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500715    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-system-cni-dir\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.501444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500731    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/88f6921c-8fc8-45b4-b4ac-368be491377b-multus-daemon-config\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.501444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500754    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f4c4cf1-bf9c-499d-b25e-fe93588e6c58-host\") pod \"node-ca-v47sj\" (UID: \"0f4c4cf1-bf9c-499d-b25e-fe93588e6c58\") " pod="openshift-image-registry/node-ca-v47sj"
Sep 09 03:13:05.501444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500760    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e455be0-bf16-41ad-a378-7621bcf270c8-env-overrides\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.501444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-run-netns\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.501444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500812    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9vc5\" (UniqueName: \"kubernetes.io/projected/88f6921c-8fc8-45b4-b4ac-368be491377b-kube-api-access-b9vc5\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.501444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500837    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-run-openvswitch\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.501444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500868    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvbn6\" (UniqueName: \"kubernetes.io/projected/4b3084e8-6418-4139-9623-5528c6b2cef3-kube-api-access-hvbn6\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.501444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500917    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-host-var-lib-kubelet\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.501444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500942    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-host-run-multus-certs\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.501444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500636    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-multus-socket-dir-parent\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.501444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500970    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b3084e8-6418-4139-9623-5528c6b2cef3-system-cni-dir\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.501444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.500995    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b3084e8-6418-4139-9623-5528c6b2cef3-cnibin\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.501444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501019    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-cnibin\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.501444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501029    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-system-cni-dir\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.502388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501045    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-kubelet\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.502388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501072    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-var-lib-openvswitch\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.502388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501091    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f4c4cf1-bf9c-499d-b25e-fe93588e6c58-host\") pod \"node-ca-v47sj\" (UID: \"0f4c4cf1-bf9c-499d-b25e-fe93588e6c58\") " pod="openshift-image-registry/node-ca-v47sj"
Sep 09 03:13:05.502388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501102    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-multus-cni-dir\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.502388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501131    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-host-run-k8s-cni-cncf-io\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.502388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501135    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-run-netns\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.502388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501156    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f4c4cf1-bf9c-499d-b25e-fe93588e6c58-serviceca\") pod \"node-ca-v47sj\" (UID: \"0f4c4cf1-bf9c-499d-b25e-fe93588e6c58\") " pod="openshift-image-registry/node-ca-v47sj"
Sep 09 03:13:05.502388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501181    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-log-socket\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.502388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501205    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88f6921c-8fc8-45b4-b4ac-368be491377b-cni-binary-copy\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.502388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501212    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e455be0-bf16-41ad-a378-7621bcf270c8-ovnkube-script-lib\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.502388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501232    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.502388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501229    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e455be0-bf16-41ad-a378-7621bcf270c8-ovnkube-config\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.502388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501285    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-run-openvswitch\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.502388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501291    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-host-var-lib-kubelet\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.502388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501291    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/88f6921c-8fc8-45b4-b4ac-368be491377b-multus-daemon-config\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.502388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501333    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b3084e8-6418-4139-9623-5528c6b2cef3-cnibin\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.502388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501342    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-run-ovn\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.502388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501356    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-kubelet\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.503309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501361    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-host-run-multus-certs\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.503309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501361    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b3084e8-6418-4139-9623-5528c6b2cef3-system-cni-dir\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.503309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501383    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b3084e8-6418-4139-9623-5528c6b2cef3-cni-binary-copy\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.503309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501388    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-cnibin\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.503309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501406    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-run-ovn\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.503309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501410    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-hostroot\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.503309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501430    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-multus-cni-dir\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.503309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-log-socket\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.503309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-multus-conf-dir\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.503309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501448    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-hostroot\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.503309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501483    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-host-run-k8s-cni-cncf-io\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.503309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501506    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-multus-conf-dir\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.503309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501305    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-var-lib-openvswitch\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.503309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501570    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74dqp\" (UniqueName: \"kubernetes.io/projected/0f4c4cf1-bf9c-499d-b25e-fe93588e6c58-kube-api-access-74dqp\") pod \"node-ca-v47sj\" (UID: \"0f4c4cf1-bf9c-499d-b25e-fe93588e6c58\") " pod="openshift-image-registry/node-ca-v47sj"
Sep 09 03:13:05.503309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501602    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-run-systemd\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.503309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501630    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b3084e8-6418-4139-9623-5528c6b2cef3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.503309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501679    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-etc-kubernetes\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.503309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-slash\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.504237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501730    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-node-log\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.504237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501755    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-cni-netd\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.504237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501784    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b3084e8-6418-4139-9623-5528c6b2cef3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.504237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.501994    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88f6921c-8fc8-45b4-b4ac-368be491377b-cni-binary-copy\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.504237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.502043    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f4c4cf1-bf9c-499d-b25e-fe93588e6c58-serviceca\") pod \"node-ca-v47sj\" (UID: \"0f4c4cf1-bf9c-499d-b25e-fe93588e6c58\") " pod="openshift-image-registry/node-ca-v47sj"
Sep 09 03:13:05.504237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.502051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f6921c-8fc8-45b4-b4ac-368be491377b-etc-kubernetes\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.504237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.502093    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-run-systemd\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.504237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.502105    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b3084e8-6418-4139-9623-5528c6b2cef3-cni-binary-copy\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.504237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.502107    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b3084e8-6418-4139-9623-5528c6b2cef3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.504237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.502151    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.504237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.502169    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-node-log\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.504237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.502214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-cni-netd\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.504237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.502233    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-host-slash\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.504237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.502232    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b3084e8-6418-4139-9623-5528c6b2cef3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.504237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.502344    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e455be0-bf16-41ad-a378-7621bcf270c8-systemd-units\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.504237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.502345    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b3084e8-6418-4139-9623-5528c6b2cef3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.504964 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.504852    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e455be0-bf16-41ad-a378-7621bcf270c8-ovn-node-metrics-cert\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.510911 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:05.510834    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:13:05.510911 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:05.510860    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:13:05.510911 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:05.510873    2560 projected.go:196] Error preparing data for projected volume kube-api-access-jl5z2 for pod openshift-network-diagnostics/network-check-target-tls5w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:13:05.511111 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:05.510940    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2 podName:d539727e-a1d4-43f5-be9e-84c6611912be nodeName:}" failed. No retries permitted until 2026-09-09 03:13:06.010922672 +0000 UTC m=+3.147670523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jl5z2" (UniqueName: "kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2") pod "network-check-target-tls5w" (UID: "d539727e-a1d4-43f5-be9e-84c6611912be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:13:05.514153 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.514128    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp57z\" (UniqueName: \"kubernetes.io/projected/f8508862-2ffb-4bd8-bb4f-67cd69a78075-kube-api-access-zp57z\") pod \"network-metrics-daemon-hlmbb\" (UID: \"f8508862-2ffb-4bd8-bb4f-67cd69a78075\") " pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:05.514483 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.514459    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7w7g\" (UniqueName: \"kubernetes.io/projected/7e455be0-bf16-41ad-a378-7621bcf270c8-kube-api-access-n7w7g\") pod \"ovnkube-node-cjmpt\" (UID: \"7e455be0-bf16-41ad-a378-7621bcf270c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.514829 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.514799    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9vc5\" (UniqueName: \"kubernetes.io/projected/88f6921c-8fc8-45b4-b4ac-368be491377b-kube-api-access-b9vc5\") pod \"multus-zxq68\" (UID: \"88f6921c-8fc8-45b4-b4ac-368be491377b\") " pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.515530 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.515495    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74dqp\" (UniqueName: \"kubernetes.io/projected/0f4c4cf1-bf9c-499d-b25e-fe93588e6c58-kube-api-access-74dqp\") pod \"node-ca-v47sj\" (UID: \"0f4c4cf1-bf9c-499d-b25e-fe93588e6c58\") " pod="openshift-image-registry/node-ca-v47sj"
Sep 09 03:13:05.515869 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.515848    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvbn6\" (UniqueName: \"kubernetes.io/projected/4b3084e8-6418-4139-9623-5528c6b2cef3-kube-api-access-hvbn6\") pod \"multus-additional-cni-plugins-sbbhj\" (UID: \"4b3084e8-6418-4139-9623-5528c6b2cef3\") " pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.586902 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.586867    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sbmb7"
Sep 09 03:13:05.601721 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.601697    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rlqww"
Sep 09 03:13:05.611820 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.611800    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7nq7n"
Sep 09 03:13:05.617738 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.617704    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7xspc"
Sep 09 03:13:05.626644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.626622    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb"
Sep 09 03:13:05.635663 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.635644    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:05.643664 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.643643    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sbbhj"
Sep 09 03:13:05.651698 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.651657    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zxq68"
Sep 09 03:13:05.659514 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.659489    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v47sj"
Sep 09 03:13:05.905376 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:05.905289    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret\") pod \"global-pull-secret-syncer-v89sv\" (UID: \"422df607-4fd9-4edd-ad99-7b6214c7d970\") " pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:05.905536 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:05.905415    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:13:05.905536 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:05.905476    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret podName:422df607-4fd9-4edd-ad99-7b6214c7d970 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:06.905460163 +0000 UTC m=+4.042208018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret") pod "global-pull-secret-syncer-v89sv" (UID: "422df607-4fd9-4edd-ad99-7b6214c7d970") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:13:06.006662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:06.006606    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs\") pod \"network-metrics-daemon-hlmbb\" (UID: \"f8508862-2ffb-4bd8-bb4f-67cd69a78075\") " pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:06.006854 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:06.006786    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:13:06.006854 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:06.006854    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs podName:f8508862-2ffb-4bd8-bb4f-67cd69a78075 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:07.006837581 +0000 UTC m=+4.143585435 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs") pod "network-metrics-daemon-hlmbb" (UID: "f8508862-2ffb-4bd8-bb4f-67cd69a78075") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:13:06.107523 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:06.107482    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl5z2\" (UniqueName: \"kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2\") pod \"network-check-target-tls5w\" (UID: \"d539727e-a1d4-43f5-be9e-84c6611912be\") " pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:06.107746 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:06.107722    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:13:06.107843 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:06.107754    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:13:06.107843 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:06.107769    2560 projected.go:196] Error preparing data for projected volume kube-api-access-jl5z2 for pod openshift-network-diagnostics/network-check-target-tls5w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:13:06.107843 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:06.107839    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2 podName:d539727e-a1d4-43f5-be9e-84c6611912be nodeName:}" failed. No retries permitted until 2026-09-09 03:13:07.107819455 +0000 UTC m=+4.244567301 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jl5z2" (UniqueName: "kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2") pod "network-check-target-tls5w" (UID: "d539727e-a1d4-43f5-be9e-84c6611912be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:13:06.325122 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:06.325019    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 03:08:05 +0000 UTC" deadline="2028-04-03 07:27:39.68482305 +0000 UTC"
Sep 09 03:13:06.325122 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:06.325067    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13732h14m33.359758826s"
Sep 09 03:13:06.566210 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:06.566048    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79f2fe7_aee6_4fe3_9834_1495344fc754.slice/crio-d62c4bf3672758edad10aaf973fa07bb70d8812611b299a90d34a86800c54df4 WatchSource:0}: Error finding container d62c4bf3672758edad10aaf973fa07bb70d8812611b299a90d34a86800c54df4: Status 404 returned error can't find the container with id d62c4bf3672758edad10aaf973fa07bb70d8812611b299a90d34a86800c54df4
Sep 09 03:13:06.570040 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:06.570012    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode698c58a_191f_485e_8258_fae42a7238e3.slice/crio-d7a3b82983e1d8f2a1d338e0ebfba5705961d58c96eba8d7a2232738aa9a7aea WatchSource:0}: Error finding container d7a3b82983e1d8f2a1d338e0ebfba5705961d58c96eba8d7a2232738aa9a7aea: Status 404 returned error can't find the container with id d7a3b82983e1d8f2a1d338e0ebfba5705961d58c96eba8d7a2232738aa9a7aea
Sep 09 03:13:06.571183 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:06.571136    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f6921c_8fc8_45b4_b4ac_368be491377b.slice/crio-ee16fa0927def96b610ac652c3103ad7d24f09b8f11b628215ac0a50b8d5adc6 WatchSource:0}: Error finding container ee16fa0927def96b610ac652c3103ad7d24f09b8f11b628215ac0a50b8d5adc6: Status 404 returned error can't find the container with id ee16fa0927def96b610ac652c3103ad7d24f09b8f11b628215ac0a50b8d5adc6
Sep 09 03:13:06.572182 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:06.572156    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965c7f4b_4baa_474b_a29e_8eb699e4752e.slice/crio-d806fb9553e3dbf6fe2cd783902151f448a43de30c8e3084457c4a570b6edbb7 WatchSource:0}: Error finding container d806fb9553e3dbf6fe2cd783902151f448a43de30c8e3084457c4a570b6edbb7: Status 404 returned error can't find the container with id d806fb9553e3dbf6fe2cd783902151f448a43de30c8e3084457c4a570b6edbb7
Sep 09 03:13:06.573325 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:06.573292    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8465770f_1ae2_4566_8c6d_12cc331da1ec.slice/crio-ad5758e81224468a9d6757ea59798efc6ee6cd718a6a44fcb5b6456298dc912c WatchSource:0}: Error finding container ad5758e81224468a9d6757ea59798efc6ee6cd718a6a44fcb5b6456298dc912c: Status 404 returned error can't find the container with id ad5758e81224468a9d6757ea59798efc6ee6cd718a6a44fcb5b6456298dc912c
Sep 09 03:13:06.575091 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:06.575065    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3084e8_6418_4139_9623_5528c6b2cef3.slice/crio-c1f54ee25f88118e3941054efcad9f8ef5ff6763e299d9589687d3b6fc9ebe80 WatchSource:0}: Error finding container c1f54ee25f88118e3941054efcad9f8ef5ff6763e299d9589687d3b6fc9ebe80: Status 404 returned error can't find the container with id c1f54ee25f88118e3941054efcad9f8ef5ff6763e299d9589687d3b6fc9ebe80
Sep 09 03:13:06.577454 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:06.576511    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4c4cf1_bf9c_499d_b25e_fe93588e6c58.slice/crio-1b464bfd2fef22fb8e4efd8c65529aababb895ac1f0a0fd464380e67184b07e6 WatchSource:0}: Error finding container 1b464bfd2fef22fb8e4efd8c65529aababb895ac1f0a0fd464380e67184b07e6: Status 404 returned error can't find the container with id 1b464bfd2fef22fb8e4efd8c65529aababb895ac1f0a0fd464380e67184b07e6
Sep 09 03:13:06.580176 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:06.579211    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef2fdc5_08ed_43a3_b9c1_b8b55e6444c5.slice/crio-81bf36313d2870c22981b99a055a5124ce5890623081fce92088756203de5a3c WatchSource:0}: Error finding container 81bf36313d2870c22981b99a055a5124ce5890623081fce92088756203de5a3c: Status 404 returned error can't find the container with id 81bf36313d2870c22981b99a055a5124ce5890623081fce92088756203de5a3c
Sep 09 03:13:06.580176 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:06.579506    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e455be0_bf16_41ad_a378_7621bcf270c8.slice/crio-cc64718c212f88f008ced4af106c79f87cfd7b85dfeb6ab63c3017ea1977ec5e WatchSource:0}: Error finding container cc64718c212f88f008ced4af106c79f87cfd7b85dfeb6ab63c3017ea1977ec5e: Status 404 returned error can't find the container with id cc64718c212f88f008ced4af106c79f87cfd7b85dfeb6ab63c3017ea1977ec5e
Sep 09 03:13:06.915821 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:06.915581    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret\") pod \"global-pull-secret-syncer-v89sv\" (UID: \"422df607-4fd9-4edd-ad99-7b6214c7d970\") " pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:06.915932 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:06.915800    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:13:06.915932 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:06.915897    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret podName:422df607-4fd9-4edd-ad99-7b6214c7d970 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:08.915876617 +0000 UTC m=+6.052624477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret") pod "global-pull-secret-syncer-v89sv" (UID: "422df607-4fd9-4edd-ad99-7b6214c7d970") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:13:06.936072 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:06.936050    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:13:07.016883 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.016848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs\") pod \"network-metrics-daemon-hlmbb\" (UID: \"f8508862-2ffb-4bd8-bb4f-67cd69a78075\") " pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:07.017058 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:07.017009    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:13:07.017125 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:07.017088    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs podName:f8508862-2ffb-4bd8-bb4f-67cd69a78075 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:09.017067581 +0000 UTC m=+6.153815434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs") pod "network-metrics-daemon-hlmbb" (UID: "f8508862-2ffb-4bd8-bb4f-67cd69a78075") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:13:07.118261 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.118135    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl5z2\" (UniqueName: \"kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2\") pod \"network-check-target-tls5w\" (UID: \"d539727e-a1d4-43f5-be9e-84c6611912be\") " pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:07.118433 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:07.118377    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:13:07.118433 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:07.118405    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:13:07.118433 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:07.118420    2560 projected.go:196] Error preparing data for projected volume kube-api-access-jl5z2 for pod openshift-network-diagnostics/network-check-target-tls5w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:13:07.118621 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:07.118482    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2 podName:d539727e-a1d4-43f5-be9e-84c6611912be nodeName:}" failed. No retries permitted until 2026-09-09 03:13:09.118463246 +0000 UTC m=+6.255211099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jl5z2" (UniqueName: "kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2") pod "network-check-target-tls5w" (UID: "d539727e-a1d4-43f5-be9e-84c6611912be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:13:07.326205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.326157    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 03:08:05 +0000 UTC" deadline="2028-04-14 07:29:16.473631411 +0000 UTC"
Sep 09 03:13:07.326877 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.326243    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13996h16m9.147414234s"
Sep 09 03:13:07.355829 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.355319    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:07.355829 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:07.355448    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tls5w" podUID="d539727e-a1d4-43f5-be9e-84c6611912be"
Sep 09 03:13:07.356051 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.355896    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:07.356051 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:07.356026    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:13:07.356164 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.356058    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:07.356164 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:07.356137    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v89sv" podUID="422df607-4fd9-4edd-ad99-7b6214c7d970"
Sep 09 03:13:07.377771 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.377697    2560 generic.go:356] "Generic (PLEG): container finished" podID="2c3c6a94ac7c6993af73a9ccfb982bee" containerID="8c2ae81e0f92750bbca46003d989b47c02abb0aca122c9cebbd9c8e771f1f5c7" exitCode=0
Sep 09 03:13:07.377910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.377786    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal" event={"ID":"2c3c6a94ac7c6993af73a9ccfb982bee","Type":"ContainerDied","Data":"8c2ae81e0f92750bbca46003d989b47c02abb0aca122c9cebbd9c8e771f1f5c7"}
Sep 09 03:13:07.386109 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.385986    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt" event={"ID":"7e455be0-bf16-41ad-a378-7621bcf270c8","Type":"ContainerStarted","Data":"cc64718c212f88f008ced4af106c79f87cfd7b85dfeb6ab63c3017ea1977ec5e"}
Sep 09 03:13:07.392292 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.392225    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sbmb7" event={"ID":"def2fdc5-08ed-43a3-b9c1-b8b55e6444c5","Type":"ContainerStarted","Data":"81bf36313d2870c22981b99a055a5124ce5890623081fce92088756203de5a3c"}
Sep 09 03:13:07.406392 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.406361    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v47sj" event={"ID":"0f4c4cf1-bf9c-499d-b25e-fe93588e6c58","Type":"ContainerStarted","Data":"1b464bfd2fef22fb8e4efd8c65529aababb895ac1f0a0fd464380e67184b07e6"}
Sep 09 03:13:07.422254 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.422212    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7xspc" event={"ID":"8465770f-1ae2-4566-8c6d-12cc331da1ec","Type":"ContainerStarted","Data":"ad5758e81224468a9d6757ea59798efc6ee6cd718a6a44fcb5b6456298dc912c"}
Sep 09 03:13:07.424931 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.424904    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zxq68" event={"ID":"88f6921c-8fc8-45b4-b4ac-368be491377b","Type":"ContainerStarted","Data":"ee16fa0927def96b610ac652c3103ad7d24f09b8f11b628215ac0a50b8d5adc6"}
Sep 09 03:13:07.429428 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.429399    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7nq7n" event={"ID":"e79f2fe7-aee6-4fe3-9834-1495344fc754","Type":"ContainerStarted","Data":"d62c4bf3672758edad10aaf973fa07bb70d8812611b299a90d34a86800c54df4"}
Sep 09 03:13:07.447843 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.447809    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal" event={"ID":"50ae6cedb6cbe5050a1216b96fc89c4f","Type":"ContainerStarted","Data":"c5e8820367b46ffd97581161ad80329f77e03d127c6c02f876da159a9fe04c20"}
Sep 09 03:13:07.457314 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.457284    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sbbhj" event={"ID":"4b3084e8-6418-4139-9623-5528c6b2cef3","Type":"ContainerStarted","Data":"c1f54ee25f88118e3941054efcad9f8ef5ff6763e299d9589687d3b6fc9ebe80"}
Sep 09 03:13:07.465453 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.465403    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rlqww" event={"ID":"965c7f4b-4baa-474b-a29e-8eb699e4752e","Type":"ContainerStarted","Data":"d806fb9553e3dbf6fe2cd783902151f448a43de30c8e3084457c4a570b6edbb7"}
Sep 09 03:13:07.477823 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:07.477794    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb" event={"ID":"e698c58a-191f-485e-8258-fae42a7238e3","Type":"ContainerStarted","Data":"d7a3b82983e1d8f2a1d338e0ebfba5705961d58c96eba8d7a2232738aa9a7aea"}
Sep 09 03:13:08.506503 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:08.506364    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal" event={"ID":"2c3c6a94ac7c6993af73a9ccfb982bee","Type":"ContainerStarted","Data":"8b0e830e62d2d132306657a47ed04df4288b5a41a75deea71da1beaf4515ef19"}
Sep 09 03:13:08.524786 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:08.523276    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal" podStartSLOduration=3.523256009 podStartE2EDuration="3.523256009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:08.522787875 +0000 UTC m=+5.659535763" watchObservedRunningTime="2026-09-09 03:13:08.523256009 +0000 UTC m=+5.660003878"
Sep 09 03:13:08.524786 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:08.524213    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal" podStartSLOduration=3.524200478 podStartE2EDuration="3.524200478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:07.464380309 +0000 UTC m=+4.601128180" watchObservedRunningTime="2026-09-09 03:13:08.524200478 +0000 UTC m=+5.660948349"
Sep 09 03:13:08.936319 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:08.935636    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret\") pod \"global-pull-secret-syncer-v89sv\" (UID: \"422df607-4fd9-4edd-ad99-7b6214c7d970\") " pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:08.936319 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:08.935787    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:13:08.936319 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:08.935859    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret podName:422df607-4fd9-4edd-ad99-7b6214c7d970 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:12.935838894 +0000 UTC m=+10.072586744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret") pod "global-pull-secret-syncer-v89sv" (UID: "422df607-4fd9-4edd-ad99-7b6214c7d970") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:13:09.037494 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:09.036683    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs\") pod \"network-metrics-daemon-hlmbb\" (UID: \"f8508862-2ffb-4bd8-bb4f-67cd69a78075\") " pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:09.037494 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:09.036978    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:13:09.037494 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:09.037047    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs podName:f8508862-2ffb-4bd8-bb4f-67cd69a78075 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:13.037027588 +0000 UTC m=+10.173775435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs") pod "network-metrics-daemon-hlmbb" (UID: "f8508862-2ffb-4bd8-bb4f-67cd69a78075") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:13:09.137842 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:09.137802    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl5z2\" (UniqueName: \"kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2\") pod \"network-check-target-tls5w\" (UID: \"d539727e-a1d4-43f5-be9e-84c6611912be\") " pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:09.138137 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:09.138114    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:13:09.138221 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:09.138150    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:13:09.138221 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:09.138164    2560 projected.go:196] Error preparing data for projected volume kube-api-access-jl5z2 for pod openshift-network-diagnostics/network-check-target-tls5w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:13:09.138333 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:09.138230    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2 podName:d539727e-a1d4-43f5-be9e-84c6611912be nodeName:}" failed. No retries permitted until 2026-09-09 03:13:13.138209493 +0000 UTC m=+10.274957341 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jl5z2" (UniqueName: "kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2") pod "network-check-target-tls5w" (UID: "d539727e-a1d4-43f5-be9e-84c6611912be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:13:09.353881 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:09.353793    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:09.354060 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:09.353936    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tls5w" podUID="d539727e-a1d4-43f5-be9e-84c6611912be"
Sep 09 03:13:09.354398 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:09.354212    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:09.354398 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:09.354345    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v89sv" podUID="422df607-4fd9-4edd-ad99-7b6214c7d970"
Sep 09 03:13:09.354588 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:09.354421    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:09.354588 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:09.354531    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:13:11.354792 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:11.354756    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:11.355222 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:11.354905    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:13:11.355222 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:11.355135    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:11.355329 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:11.355254    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v89sv" podUID="422df607-4fd9-4edd-ad99-7b6214c7d970"
Sep 09 03:13:11.355385 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:11.355327    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:11.355466 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:11.355442    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tls5w" podUID="d539727e-a1d4-43f5-be9e-84c6611912be"
Sep 09 03:13:12.969917 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:12.969876    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret\") pod \"global-pull-secret-syncer-v89sv\" (UID: \"422df607-4fd9-4edd-ad99-7b6214c7d970\") " pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:12.970388 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:12.970024    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:13:12.970388 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:12.970113    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret podName:422df607-4fd9-4edd-ad99-7b6214c7d970 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:20.970091488 +0000 UTC m=+18.106839351 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret") pod "global-pull-secret-syncer-v89sv" (UID: "422df607-4fd9-4edd-ad99-7b6214c7d970") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:13:13.070620 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:13.070583    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs\") pod \"network-metrics-daemon-hlmbb\" (UID: \"f8508862-2ffb-4bd8-bb4f-67cd69a78075\") " pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:13.070782 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:13.070739    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:13:13.070866 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:13.070814    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs podName:f8508862-2ffb-4bd8-bb4f-67cd69a78075 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:21.070792668 +0000 UTC m=+18.207540517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs") pod "network-metrics-daemon-hlmbb" (UID: "f8508862-2ffb-4bd8-bb4f-67cd69a78075") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:13:13.171704 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:13.171665    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl5z2\" (UniqueName: \"kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2\") pod \"network-check-target-tls5w\" (UID: \"d539727e-a1d4-43f5-be9e-84c6611912be\") " pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:13.171939 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:13.171854    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:13:13.171939 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:13.171884    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:13:13.171939 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:13.171896    2560 projected.go:196] Error preparing data for projected volume kube-api-access-jl5z2 for pod openshift-network-diagnostics/network-check-target-tls5w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:13:13.172112 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:13.171970    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2 podName:d539727e-a1d4-43f5-be9e-84c6611912be nodeName:}" failed. No retries permitted until 2026-09-09 03:13:21.171940599 +0000 UTC m=+18.308688443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jl5z2" (UniqueName: "kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2") pod "network-check-target-tls5w" (UID: "d539727e-a1d4-43f5-be9e-84c6611912be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:13:13.355082 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:13.355004    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:13.355236 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:13.355124    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v89sv" podUID="422df607-4fd9-4edd-ad99-7b6214c7d970"
Sep 09 03:13:13.355607 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:13.355534    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:13.355739 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:13.355631    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tls5w" podUID="d539727e-a1d4-43f5-be9e-84c6611912be"
Sep 09 03:13:13.355819 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:13.355782    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:13.355929 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:13.355881    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:13:15.352910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:15.352857    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:15.353479 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:15.352990    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v89sv" podUID="422df607-4fd9-4edd-ad99-7b6214c7d970"
Sep 09 03:13:15.353479 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:15.353371    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:15.353479 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:15.353457    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tls5w" podUID="d539727e-a1d4-43f5-be9e-84c6611912be"
Sep 09 03:13:15.353749 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:15.353625    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:15.353803 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:15.353746    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:13:17.352505 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:17.352475    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:17.352984 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:17.352518    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:17.352984 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:17.352474    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:17.352984 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:17.352587    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v89sv" podUID="422df607-4fd9-4edd-ad99-7b6214c7d970"
Sep 09 03:13:17.352984 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:17.352704    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tls5w" podUID="d539727e-a1d4-43f5-be9e-84c6611912be"
Sep 09 03:13:17.352984 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:17.352781    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:13:19.353326 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:19.353291    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:19.353326 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:19.353332    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:19.353931 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:19.353298    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:19.353931 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:19.353430    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:13:19.353931 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:19.353528    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v89sv" podUID="422df607-4fd9-4edd-ad99-7b6214c7d970"
Sep 09 03:13:19.353931 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:19.353633    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tls5w" podUID="d539727e-a1d4-43f5-be9e-84c6611912be"
Sep 09 03:13:21.030572 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:21.030533    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret\") pod \"global-pull-secret-syncer-v89sv\" (UID: \"422df607-4fd9-4edd-ad99-7b6214c7d970\") " pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:21.031015 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:21.030677    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:13:21.031015 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:21.030752    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret podName:422df607-4fd9-4edd-ad99-7b6214c7d970 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:37.030732061 +0000 UTC m=+34.167479907 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret") pod "global-pull-secret-syncer-v89sv" (UID: "422df607-4fd9-4edd-ad99-7b6214c7d970") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:13:21.131309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:21.131278    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs\") pod \"network-metrics-daemon-hlmbb\" (UID: \"f8508862-2ffb-4bd8-bb4f-67cd69a78075\") " pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:21.131464 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:21.131447    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:13:21.131537 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:21.131525    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs podName:f8508862-2ffb-4bd8-bb4f-67cd69a78075 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:37.131504395 +0000 UTC m=+34.268252258 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs") pod "network-metrics-daemon-hlmbb" (UID: "f8508862-2ffb-4bd8-bb4f-67cd69a78075") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:13:21.231823 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:21.231783    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl5z2\" (UniqueName: \"kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2\") pod \"network-check-target-tls5w\" (UID: \"d539727e-a1d4-43f5-be9e-84c6611912be\") " pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:21.231994 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:21.231973    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:13:21.232053 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:21.232003    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:13:21.232053 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:21.232016    2560 projected.go:196] Error preparing data for projected volume kube-api-access-jl5z2 for pod openshift-network-diagnostics/network-check-target-tls5w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:13:21.232154 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:21.232083    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2 podName:d539727e-a1d4-43f5-be9e-84c6611912be nodeName:}" failed. No retries permitted until 2026-09-09 03:13:37.232062645 +0000 UTC m=+34.368810510 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jl5z2" (UniqueName: "kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2") pod "network-check-target-tls5w" (UID: "d539727e-a1d4-43f5-be9e-84c6611912be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:13:21.352525 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:21.352489    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:21.352693 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:21.352536    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:21.352693 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:21.352640    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:21.352693 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:21.352658    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:13:21.352861 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:21.352729    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tls5w" podUID="d539727e-a1d4-43f5-be9e-84c6611912be"
Sep 09 03:13:21.353547 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:21.352811    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v89sv" podUID="422df607-4fd9-4edd-ad99-7b6214c7d970"
Sep 09 03:13:23.353405 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:23.353372    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:23.353855 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:23.353478    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v89sv" podUID="422df607-4fd9-4edd-ad99-7b6214c7d970"
Sep 09 03:13:23.353855 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:23.353579    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:23.353855 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:23.353676    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tls5w" podUID="d539727e-a1d4-43f5-be9e-84c6611912be"
Sep 09 03:13:23.353855 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:23.353728    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:23.353855 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:23.353802    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:13:25.353484 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.353289    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:25.354450 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.353288    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:25.354450 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:25.353558    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v89sv" podUID="422df607-4fd9-4edd-ad99-7b6214c7d970"
Sep 09 03:13:25.354450 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.353350    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:25.354450 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:25.353638    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tls5w" podUID="d539727e-a1d4-43f5-be9e-84c6611912be"
Sep 09 03:13:25.354450 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:25.353711    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:13:25.561406 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.561371    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb" event={"ID":"e698c58a-191f-485e-8258-fae42a7238e3","Type":"ContainerStarted","Data":"af9e84b4f863346917d00cd216bc7ac7bb7311c617de79ace5bd5e4ac40ce41c"}
Sep 09 03:13:25.564968 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.564939    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt" event={"ID":"7e455be0-bf16-41ad-a378-7621bcf270c8","Type":"ContainerStarted","Data":"f408340683a9452a12efa5211e628d19510e402ba8a1e1cb7b0395b5d42cf225"}
Sep 09 03:13:25.565142 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.565028    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt" event={"ID":"7e455be0-bf16-41ad-a378-7621bcf270c8","Type":"ContainerStarted","Data":"5e329d087d0bc5ca780810831278679cd93f8f432cc1259fb5637cb1155a5191"}
Sep 09 03:13:25.565142 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.565084    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt" event={"ID":"7e455be0-bf16-41ad-a378-7621bcf270c8","Type":"ContainerStarted","Data":"29ccb55eb2c45d3f944e8e2634f355d62b83514e0b099f5c256a08d864c21205"}
Sep 09 03:13:25.565142 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.565130    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt" event={"ID":"7e455be0-bf16-41ad-a378-7621bcf270c8","Type":"ContainerStarted","Data":"def964367ac1bcafa9855adb4674f56f555ee523b7f48901ff43a87becb08128"}
Sep 09 03:13:25.565280 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.565178    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt" event={"ID":"7e455be0-bf16-41ad-a378-7621bcf270c8","Type":"ContainerStarted","Data":"ec7a38090e95d61eb9691b9a3cb67ba0a1e1181534d0525becbfe2153331d2be"}
Sep 09 03:13:25.565280 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.565224    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt" event={"ID":"7e455be0-bf16-41ad-a378-7621bcf270c8","Type":"ContainerStarted","Data":"143d0cda688e0166616923f28da7fb3334081b565fd33080967c68df07261436"}
Sep 09 03:13:25.566578 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.566541    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sbmb7" event={"ID":"def2fdc5-08ed-43a3-b9c1-b8b55e6444c5","Type":"ContainerStarted","Data":"42f0f69736261230b6fbacd7064cc11850c0fff2138f0141d3be8637f98dd9c2"}
Sep 09 03:13:25.568219 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.568197    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v47sj" event={"ID":"0f4c4cf1-bf9c-499d-b25e-fe93588e6c58","Type":"ContainerStarted","Data":"292700a80026772b475f1ed351f74e5665965fb285f182475485c2f3bf4c287b"}
Sep 09 03:13:25.569856 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.569819    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7xspc" event={"ID":"8465770f-1ae2-4566-8c6d-12cc331da1ec","Type":"ContainerStarted","Data":"ebf5c12b7511b135e7ca9390998aaa0b4b6613cde039ce35aadccbcf10339ee0"}
Sep 09 03:13:25.572272 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.572246    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zxq68" event={"ID":"88f6921c-8fc8-45b4-b4ac-368be491377b","Type":"ContainerStarted","Data":"d89e85ea1f4b1d633405998fc68dd7ff7dcf76e26d52d35b93eeab3a6499d731"}
Sep 09 03:13:25.574532 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.574507    2560 generic.go:356] "Generic (PLEG): container finished" podID="4b3084e8-6418-4139-9623-5528c6b2cef3" containerID="d8bda698a564f8d0648933d5e371ac8bc7dc7866d4289e2c302fa60399a289e2" exitCode=0
Sep 09 03:13:25.574658 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.574584    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sbbhj" event={"ID":"4b3084e8-6418-4139-9623-5528c6b2cef3","Type":"ContainerDied","Data":"d8bda698a564f8d0648933d5e371ac8bc7dc7866d4289e2c302fa60399a289e2"}
Sep 09 03:13:25.576332 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.576306    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rlqww" event={"ID":"965c7f4b-4baa-474b-a29e-8eb699e4752e","Type":"ContainerStarted","Data":"22c5a4d99b34f3830b219fb8cf59a10b40e3469394f1b4531b41a45fb691a103"}
Sep 09 03:13:25.604105 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.602394    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sbmb7" podStartSLOduration=3.728787672 podStartE2EDuration="21.60237675s" totalImagesPullingTime="17.873589078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:25.581748049 +0000 UTC m=+22.718495914" watchObservedRunningTime="2026-09-09 03:13:25.60237675 +0000 UTC m=+22.739124626"
Sep 09 03:13:25.618096 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.618061    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zxq68" podStartSLOduration=3.686029641 podStartE2EDuration="21.618048331s" totalImagesPullingTime="17.93201869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:25.6180332 +0000 UTC m=+22.754781064" watchObservedRunningTime="2026-09-09 03:13:25.618048331 +0000 UTC m=+22.754796173"
Sep 09 03:13:25.649503 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.649459    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rlqww" podStartSLOduration=3.75851149 podStartE2EDuration="21.649445472s" totalImagesPullingTime="17.890933982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:25.649125058 +0000 UTC m=+22.785872922" watchObservedRunningTime="2026-09-09 03:13:25.649445472 +0000 UTC m=+22.786193336"
Sep 09 03:13:25.649646 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.649545    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v47sj" podStartSLOduration=3.786136316 podStartE2EDuration="21.649541363s" totalImagesPullingTime="17.863405047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:25.631329107 +0000 UTC m=+22.768076989" watchObservedRunningTime="2026-09-09 03:13:25.649541363 +0000 UTC m=+22.786289227"
Sep 09 03:13:25.664286 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:25.664249    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7xspc" podStartSLOduration=7.652795896 podStartE2EDuration="21.664236443s" totalImagesPullingTime="14.011440547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:25.664149616 +0000 UTC m=+22.800897483" watchObservedRunningTime="2026-09-09 03:13:25.664236443 +0000 UTC m=+22.800984307"
Sep 09 03:13:26.387771 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:26.387739    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 03:13:26.580020 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:26.579934    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7nq7n" event={"ID":"e79f2fe7-aee6-4fe3-9834-1495344fc754","Type":"ContainerStarted","Data":"4afc7e8ff3e2f15e80c6da2db9aa5414d1fae0ea3391d5abc314a1208d6f7a6b"}
Sep 09 03:13:26.581663 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:26.581635    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb" event={"ID":"e698c58a-191f-485e-8258-fae42a7238e3","Type":"ContainerStarted","Data":"b8b2f074d0afbc489293069dee87fe1aa2193be2915689dd2510432608074b98"}
Sep 09 03:13:26.594362 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:26.594316    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7nq7n" podStartSLOduration=4.712120752 podStartE2EDuration="22.594302073s" totalImagesPullingTime="17.882181321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:26.593681046 +0000 UTC m=+23.730428914" watchObservedRunningTime="2026-09-09 03:13:26.594302073 +0000 UTC m=+23.731049940"
Sep 09 03:13:27.352656 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:27.352629    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:27.352811 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:27.352739    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tls5w" podUID="d539727e-a1d4-43f5-be9e-84c6611912be"
Sep 09 03:13:27.352811 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:27.352629    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:27.352929 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:27.352826    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:13:27.352929 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:27.352629    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:27.352929 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:27.352893    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v89sv" podUID="422df607-4fd9-4edd-ad99-7b6214c7d970"
Sep 09 03:13:27.359748 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:27.359660    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T03:13:26.38775701Z","UUID":"03495116-ac06-4e90-9985-4e29c35c1b5a","Handler":null,"Name":"","Endpoint":""}
Sep 09 03:13:27.361182 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:27.361159    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 03:13:27.361296 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:27.361195    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 03:13:27.589066 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:27.589034    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt" event={"ID":"7e455be0-bf16-41ad-a378-7621bcf270c8","Type":"ContainerStarted","Data":"baf6eabc8113548c81f9b3c3bc4a192011c407d393ab7e6dd2bbfe0268b50304"}
Sep 09 03:13:28.594362 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:28.594105    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb" event={"ID":"e698c58a-191f-485e-8258-fae42a7238e3","Type":"ContainerStarted","Data":"921a58b83addef039066c876c7480db60bb86e57bd05b95548283f8f26a9c1e2"}
Sep 09 03:13:28.613239 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:28.613187    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2hqb" podStartSLOduration=3.97544187 podStartE2EDuration="24.613156085s" totalImagesPullingTime="20.637714215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:13:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:28.612273715 +0000 UTC m=+25.749021580" watchObservedRunningTime="2026-09-09 03:13:28.613156085 +0000 UTC m=+25.749903950"
Sep 09 03:13:29.160335 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:29.160305    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7xspc"
Sep 09 03:13:29.160966 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:29.160946    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7xspc"
Sep 09 03:13:29.352270 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:29.352242    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:29.352439 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:29.352242    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:29.352439 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:29.352242    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:29.352439 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:29.352368    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:13:29.352619 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:29.352495    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v89sv" podUID="422df607-4fd9-4edd-ad99-7b6214c7d970"
Sep 09 03:13:29.352619 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:29.352583    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tls5w" podUID="d539727e-a1d4-43f5-be9e-84c6611912be"
Sep 09 03:13:30.603081 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:30.603056    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt" event={"ID":"7e455be0-bf16-41ad-a378-7621bcf270c8","Type":"ContainerStarted","Data":"1802514c5cf516fdde1b73ea6b980954b0d2a4e12ecba158101ed0e279706e66"}
Sep 09 03:13:30.603908 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:30.603291    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:30.618365 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:30.618341    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:30.629878 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:30.629836    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt" podStartSLOduration=8.696568451 podStartE2EDuration="26.62982176s" totalImagesPullingTime="17.933253309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:30.629413296 +0000 UTC m=+27.766161160" watchObservedRunningTime="2026-09-09 03:13:30.62982176 +0000 UTC m=+27.766569662"
Sep 09 03:13:30.964172 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:30.963915    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7xspc"
Sep 09 03:13:30.964301 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:30.964259    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 03:13:30.964656 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:30.964627    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7xspc"
Sep 09 03:13:31.353005 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:31.352977    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:31.353180 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:31.352977    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:31.353180 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:31.353074    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v89sv" podUID="422df607-4fd9-4edd-ad99-7b6214c7d970"
Sep 09 03:13:31.353180 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:31.353160    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tls5w" podUID="d539727e-a1d4-43f5-be9e-84c6611912be"
Sep 09 03:13:31.353340 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:31.353210    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:31.353340 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:31.353262    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:13:31.607493 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:31.607411    2560 generic.go:356] "Generic (PLEG): container finished" podID="4b3084e8-6418-4139-9623-5528c6b2cef3" containerID="4b937bf5d373c652f1a45565d74d86cf6735caaf180fddd7cb069de06aba6cbd" exitCode=0
Sep 09 03:13:31.607902 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:31.607497    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sbbhj" event={"ID":"4b3084e8-6418-4139-9623-5528c6b2cef3","Type":"ContainerDied","Data":"4b937bf5d373c652f1a45565d74d86cf6735caaf180fddd7cb069de06aba6cbd"}
Sep 09 03:13:31.608459 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:31.608149    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:31.608459 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:31.608201    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:31.624491 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:31.623524    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:13:32.339923 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:32.339897    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v89sv"]
Sep 09 03:13:32.340038 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:32.339994    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:32.340106 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:32.340077    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v89sv" podUID="422df607-4fd9-4edd-ad99-7b6214c7d970"
Sep 09 03:13:32.343946 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:32.343828    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tls5w"]
Sep 09 03:13:32.343946 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:32.343928    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:32.344110 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:32.344070    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tls5w" podUID="d539727e-a1d4-43f5-be9e-84c6611912be"
Sep 09 03:13:32.344589 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:32.344564    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hlmbb"]
Sep 09 03:13:32.344768 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:32.344680    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:32.344900 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:32.344768    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:13:32.611211 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:32.611182    2560 generic.go:356] "Generic (PLEG): container finished" podID="4b3084e8-6418-4139-9623-5528c6b2cef3" containerID="a038d1024a2b71f0a527fb510ba723e52e37ecc1fa353677998da60696158ac0" exitCode=0
Sep 09 03:13:32.611549 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:32.611246    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sbbhj" event={"ID":"4b3084e8-6418-4139-9623-5528c6b2cef3","Type":"ContainerDied","Data":"a038d1024a2b71f0a527fb510ba723e52e37ecc1fa353677998da60696158ac0"}
Sep 09 03:13:33.616030 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:33.615992    2560 generic.go:356] "Generic (PLEG): container finished" podID="4b3084e8-6418-4139-9623-5528c6b2cef3" containerID="fcf606249a2eb0e11d4ebf35b38b544d4e1a619cba216d1dbb91bf89e32ed493" exitCode=0
Sep 09 03:13:33.616398 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:33.616058    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sbbhj" event={"ID":"4b3084e8-6418-4139-9623-5528c6b2cef3","Type":"ContainerDied","Data":"fcf606249a2eb0e11d4ebf35b38b544d4e1a619cba216d1dbb91bf89e32ed493"}
Sep 09 03:13:34.352819 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:34.352784    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:34.352959 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:34.352785    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:34.353052 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:34.352785    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:34.353119 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:34.353044    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:13:34.353171 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:34.352899    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v89sv" podUID="422df607-4fd9-4edd-ad99-7b6214c7d970"
Sep 09 03:13:34.353171 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:34.353134    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tls5w" podUID="d539727e-a1d4-43f5-be9e-84c6611912be"
Sep 09 03:13:36.352477 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:36.352447    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:36.353007 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:36.352447    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:36.353007 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:36.352562    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tls5w" podUID="d539727e-a1d4-43f5-be9e-84c6611912be"
Sep 09 03:13:36.353007 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:36.352447    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:36.353007 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:36.352684    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v89sv" podUID="422df607-4fd9-4edd-ad99-7b6214c7d970"
Sep 09 03:13:36.353007 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:36.352759    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:13:37.042834 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.042633    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret\") pod \"global-pull-secret-syncer-v89sv\" (UID: \"422df607-4fd9-4edd-ad99-7b6214c7d970\") " pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:37.043000 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.042802    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:13:37.043000 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.042945    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret podName:422df607-4fd9-4edd-ad99-7b6214c7d970 nodeName:}" failed. No retries permitted until 2026-09-09 03:14:09.042927654 +0000 UTC m=+66.179675501 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret") pod "global-pull-secret-syncer-v89sv" (UID: "422df607-4fd9-4edd-ad99-7b6214c7d970") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:13:37.143862 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.143830    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs\") pod \"network-metrics-daemon-hlmbb\" (UID: \"f8508862-2ffb-4bd8-bb4f-67cd69a78075\") " pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:37.144043 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.143973    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:13:37.144043 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.144035    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs podName:f8508862-2ffb-4bd8-bb4f-67cd69a78075 nodeName:}" failed. No retries permitted until 2026-09-09 03:14:09.144018656 +0000 UTC m=+66.280766510 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs") pod "network-metrics-daemon-hlmbb" (UID: "f8508862-2ffb-4bd8-bb4f-67cd69a78075") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:13:37.155608 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.155570    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeReady"
Sep 09 03:13:37.155759 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.155714    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 03:13:37.187834 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.187800    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-5jzgn"]
Sep 09 03:13:37.225323 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.225255    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-94bb5b9d4-65f4d"]
Sep 09 03:13:37.225465 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.225340    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:13:37.228203 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.228179    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-gd747\""
Sep 09 03:13:37.228343 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.228216    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 09 03:13:37.228343 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.228317    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 09 03:13:37.241327 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.241301    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-5jzgn"]
Sep 09 03:13:37.241451 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.241339    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h6lb8"]
Sep 09 03:13:37.241451 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.241429    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.244270 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.244054    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dmprq\""
Sep 09 03:13:37.244270 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.244097    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 03:13:37.244270 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.244115    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 03:13:37.244270 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.244161    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 03:13:37.244270 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.244229    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl5z2\" (UniqueName: \"kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2\") pod \"network-check-target-tls5w\" (UID: \"d539727e-a1d4-43f5-be9e-84c6611912be\") " pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:37.244487 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.244403    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:13:37.244487 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.244425    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:13:37.244487 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.244436    2560 projected.go:196] Error preparing data for projected volume kube-api-access-jl5z2 for pod openshift-network-diagnostics/network-check-target-tls5w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:13:37.244487 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.244484    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2 podName:d539727e-a1d4-43f5-be9e-84c6611912be nodeName:}" failed. No retries permitted until 2026-09-09 03:14:09.244466773 +0000 UTC m=+66.381214633 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jl5z2" (UniqueName: "kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2") pod "network-check-target-tls5w" (UID: "d539727e-a1d4-43f5-be9e-84c6611912be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:13:37.250773 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.250755    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 03:13:37.257850 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.257830    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c465c"]
Sep 09 03:13:37.257962 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.257948    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h6lb8"
Sep 09 03:13:37.260579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.260564    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 03:13:37.260855 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.260835    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 03:13:37.260954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.260862    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5kdc9\""
Sep 09 03:13:37.261016 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.260837    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 03:13:37.283865 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.283843    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-94bb5b9d4-65f4d"]
Sep 09 03:13:37.283970 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.283880    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h6lb8"]
Sep 09 03:13:37.284031 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.283971    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c465c"
Sep 09 03:13:37.284031 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.284006    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c465c"]
Sep 09 03:13:37.286746 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.286496    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 03:13:37.286746 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.286530    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 03:13:37.286746 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.286586    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pxbsq\""
Sep 09 03:13:37.344580 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.344548    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnfs6\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-kube-api-access-vnfs6\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.344580 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.344578    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90fe6e96-2fae-4cc7-97fb-1d46727849ed-cert\") pod \"ingress-canary-h6lb8\" (UID: \"90fe6e96-2fae-4cc7-97fb-1d46727849ed\") " pod="openshift-ingress-canary/ingress-canary-h6lb8"
Sep 09 03:13:37.344788 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.344622    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.344788 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.344658    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-nginx-conf\") pod \"networking-console-plugin-79977f9844-5jzgn\" (UID: \"6b8b2e70-6c05-4bfb-98ce-e18701db3e63\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:13:37.344788 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.344711    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87fed466-b71f-4bc5-b0e0-447ebecbaacb-ca-trust-extracted\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.344788 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.344725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-bound-sa-token\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.344788 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.344742    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-certificates\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.344788 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.344772    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87fed466-b71f-4bc5-b0e0-447ebecbaacb-trusted-ca\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.345047 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.344794    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpxtf\" (UniqueName: \"kubernetes.io/projected/90fe6e96-2fae-4cc7-97fb-1d46727849ed-kube-api-access-mpxtf\") pod \"ingress-canary-h6lb8\" (UID: \"90fe6e96-2fae-4cc7-97fb-1d46727849ed\") " pod="openshift-ingress-canary/ingress-canary-h6lb8"
Sep 09 03:13:37.345047 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.344833    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5jzgn\" (UID: \"6b8b2e70-6c05-4bfb-98ce-e18701db3e63\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:13:37.345047 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.344890    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87fed466-b71f-4bc5-b0e0-447ebecbaacb-installation-pull-secrets\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.345047 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.344938    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87fed466-b71f-4bc5-b0e0-447ebecbaacb-image-registry-private-configuration\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.445328 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.445292    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.445850 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.445332    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-nginx-conf\") pod \"networking-console-plugin-79977f9844-5jzgn\" (UID: \"6b8b2e70-6c05-4bfb-98ce-e18701db3e63\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:13:37.445850 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.445373    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87fed466-b71f-4bc5-b0e0-447ebecbaacb-ca-trust-extracted\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.445850 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.445389    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-bound-sa-token\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.445850 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.445408    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-certificates\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.445850 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.445435    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:13:37.445850 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.445463    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:13:37.445850 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.445474    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87fed466-b71f-4bc5-b0e0-447ebecbaacb-trusted-ca\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.445850 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.445486    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-94bb5b9d4-65f4d: secret "image-registry-tls" not found
Sep 09 03:13:37.445850 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.445497    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpxtf\" (UniqueName: \"kubernetes.io/projected/90fe6e96-2fae-4cc7-97fb-1d46727849ed-kube-api-access-mpxtf\") pod \"ingress-canary-h6lb8\" (UID: \"90fe6e96-2fae-4cc7-97fb-1d46727849ed\") " pod="openshift-ingress-canary/ingress-canary-h6lb8"
Sep 09 03:13:37.445850 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.445519    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5jzgn\" (UID: \"6b8b2e70-6c05-4bfb-98ce-e18701db3e63\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:13:37.445850 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.445545    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls podName:87fed466-b71f-4bc5-b0e0-447ebecbaacb nodeName:}" failed. No retries permitted until 2026-09-09 03:13:37.945527366 +0000 UTC m=+35.082275209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls") pod "image-registry-94bb5b9d4-65f4d" (UID: "87fed466-b71f-4bc5-b0e0-447ebecbaacb") : secret "image-registry-tls" not found
Sep 09 03:13:37.445850 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.445582    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a65c3985-522f-46f4-aeb2-7f12ae347816-config-volume\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:13:37.445850 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.445606    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:13:37.445850 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.445621    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gpfg\" (UniqueName: \"kubernetes.io/projected/a65c3985-522f-46f4-aeb2-7f12ae347816-kube-api-access-9gpfg\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:13:37.445850 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.445657    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert podName:6b8b2e70-6c05-4bfb-98ce-e18701db3e63 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:37.9456408 +0000 UTC m=+35.082388643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5jzgn" (UID: "6b8b2e70-6c05-4bfb-98ce-e18701db3e63") : secret "networking-console-plugin-cert" not found
Sep 09 03:13:37.445850 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.445676    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87fed466-b71f-4bc5-b0e0-447ebecbaacb-installation-pull-secrets\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.446579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.445721    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87fed466-b71f-4bc5-b0e0-447ebecbaacb-image-registry-private-configuration\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.446579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.445767    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a65c3985-522f-46f4-aeb2-7f12ae347816-tmp-dir\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:13:37.446579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.445839    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnfs6\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-kube-api-access-vnfs6\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.446579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.445868    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90fe6e96-2fae-4cc7-97fb-1d46727849ed-cert\") pod \"ingress-canary-h6lb8\" (UID: \"90fe6e96-2fae-4cc7-97fb-1d46727849ed\") " pod="openshift-ingress-canary/ingress-canary-h6lb8"
Sep 09 03:13:37.446579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.445888    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87fed466-b71f-4bc5-b0e0-447ebecbaacb-ca-trust-extracted\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.446579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.446408    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-nginx-conf\") pod \"networking-console-plugin-79977f9844-5jzgn\" (UID: \"6b8b2e70-6c05-4bfb-98ce-e18701db3e63\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:13:37.446900 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.446819    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-certificates\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.447039 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.447013    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87fed466-b71f-4bc5-b0e0-447ebecbaacb-trusted-ca\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.450424 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.450395    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87fed466-b71f-4bc5-b0e0-447ebecbaacb-installation-pull-secrets\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.450555 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.450515    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90fe6e96-2fae-4cc7-97fb-1d46727849ed-cert\") pod \"ingress-canary-h6lb8\" (UID: \"90fe6e96-2fae-4cc7-97fb-1d46727849ed\") " pod="openshift-ingress-canary/ingress-canary-h6lb8"
Sep 09 03:13:37.451203 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.451172    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87fed466-b71f-4bc5-b0e0-447ebecbaacb-image-registry-private-configuration\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.456462 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.456430    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnfs6\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-kube-api-access-vnfs6\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.456648 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.456484    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpxtf\" (UniqueName: \"kubernetes.io/projected/90fe6e96-2fae-4cc7-97fb-1d46727849ed-kube-api-access-mpxtf\") pod \"ingress-canary-h6lb8\" (UID: \"90fe6e96-2fae-4cc7-97fb-1d46727849ed\") " pod="openshift-ingress-canary/ingress-canary-h6lb8"
Sep 09 03:13:37.456769 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.456749    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-bound-sa-token\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.546508 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.546420    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:13:37.546508 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.546508    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a65c3985-522f-46f4-aeb2-7f12ae347816-config-volume\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:13:37.546753 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.546531    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gpfg\" (UniqueName: \"kubernetes.io/projected/a65c3985-522f-46f4-aeb2-7f12ae347816-kube-api-access-9gpfg\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:13:37.546753 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.546577    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a65c3985-522f-46f4-aeb2-7f12ae347816-tmp-dir\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:13:37.546753 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.546621    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:13:37.546753 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.546708    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls podName:a65c3985-522f-46f4-aeb2-7f12ae347816 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:38.046684429 +0000 UTC m=+35.183432278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls") pod "dns-default-c465c" (UID: "a65c3985-522f-46f4-aeb2-7f12ae347816") : secret "dns-default-metrics-tls" not found
Sep 09 03:13:37.547026 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.547005    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a65c3985-522f-46f4-aeb2-7f12ae347816-tmp-dir\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:13:37.547177 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.547161    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a65c3985-522f-46f4-aeb2-7f12ae347816-config-volume\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:13:37.555799 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.555778    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gpfg\" (UniqueName: \"kubernetes.io/projected/a65c3985-522f-46f4-aeb2-7f12ae347816-kube-api-access-9gpfg\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:13:37.567674 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.567652    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h6lb8"
Sep 09 03:13:37.950305 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.950266    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5jzgn\" (UID: \"6b8b2e70-6c05-4bfb-98ce-e18701db3e63\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:13:37.950473 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:37.950374    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:37.950473 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.950421    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:13:37.950578 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.950487    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert podName:6b8b2e70-6c05-4bfb-98ce-e18701db3e63 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:38.950473348 +0000 UTC m=+36.087221190 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5jzgn" (UID: "6b8b2e70-6c05-4bfb-98ce-e18701db3e63") : secret "networking-console-plugin-cert" not found
Sep 09 03:13:37.950578 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.950505    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:13:37.950578 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.950526    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-94bb5b9d4-65f4d: secret "image-registry-tls" not found
Sep 09 03:13:37.950712 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:37.950586    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls podName:87fed466-b71f-4bc5-b0e0-447ebecbaacb nodeName:}" failed. No retries permitted until 2026-09-09 03:13:38.950568268 +0000 UTC m=+36.087316115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls") pod "image-registry-94bb5b9d4-65f4d" (UID: "87fed466-b71f-4bc5-b0e0-447ebecbaacb") : secret "image-registry-tls" not found
Sep 09 03:13:38.051127 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:38.051095    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:13:38.051276 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:38.051259    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:13:38.051348 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:38.051335    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls podName:a65c3985-522f-46f4-aeb2-7f12ae347816 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:39.051311553 +0000 UTC m=+36.188059417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls") pod "dns-default-c465c" (UID: "a65c3985-522f-46f4-aeb2-7f12ae347816") : secret "dns-default-metrics-tls" not found
Sep 09 03:13:38.352412 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:38.352384    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:13:38.352549 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:38.352385    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:13:38.352620 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:38.352385    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:13:38.356069 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:38.356046    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g4cct\""
Sep 09 03:13:38.356205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:38.356100    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 03:13:38.356205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:38.356125    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 03:13:38.356205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:38.356149    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 03:13:38.356205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:38.356188    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 03:13:38.356451 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:38.356239    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rkcrq\""
Sep 09 03:13:38.665641 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:38.665559    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h6lb8"]
Sep 09 03:13:38.668802 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:13:38.668775    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fe6e96_2fae_4cc7_97fb_1d46727849ed.slice/crio-7b592125ae1cee31a45b281f0f7a33454b334782cc5877610c98e1c6f51e6cc7 WatchSource:0}: Error finding container 7b592125ae1cee31a45b281f0f7a33454b334782cc5877610c98e1c6f51e6cc7: Status 404 returned error can't find the container with id 7b592125ae1cee31a45b281f0f7a33454b334782cc5877610c98e1c6f51e6cc7
Sep 09 03:13:38.958983 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:38.958954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:38.959187 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:38.959036    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5jzgn\" (UID: \"6b8b2e70-6c05-4bfb-98ce-e18701db3e63\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:13:38.959187 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:38.959115    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:13:38.959187 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:38.959134    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-94bb5b9d4-65f4d: secret "image-registry-tls" not found
Sep 09 03:13:38.959187 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:38.959187    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls podName:87fed466-b71f-4bc5-b0e0-447ebecbaacb nodeName:}" failed. No retries permitted until 2026-09-09 03:13:40.959171866 +0000 UTC m=+38.095919714 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls") pod "image-registry-94bb5b9d4-65f4d" (UID: "87fed466-b71f-4bc5-b0e0-447ebecbaacb") : secret "image-registry-tls" not found
Sep 09 03:13:38.959369 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:38.959191    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:13:38.959369 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:38.959255    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert podName:6b8b2e70-6c05-4bfb-98ce-e18701db3e63 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:40.959235349 +0000 UTC m=+38.095983209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5jzgn" (UID: "6b8b2e70-6c05-4bfb-98ce-e18701db3e63") : secret "networking-console-plugin-cert" not found
Sep 09 03:13:39.060326 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:39.060288    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:13:39.060478 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:39.060432    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:13:39.060525 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:39.060499    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls podName:a65c3985-522f-46f4-aeb2-7f12ae347816 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:41.060480638 +0000 UTC m=+38.197228481 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls") pod "dns-default-c465c" (UID: "a65c3985-522f-46f4-aeb2-7f12ae347816") : secret "dns-default-metrics-tls" not found
Sep 09 03:13:39.633733 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:39.633692    2560 generic.go:356] "Generic (PLEG): container finished" podID="4b3084e8-6418-4139-9623-5528c6b2cef3" containerID="d497bcc3a800f153d4923e8482db8de3f696d7845d28e3a9a6ddb79278dc8c94" exitCode=0
Sep 09 03:13:39.633909 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:39.633765    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sbbhj" event={"ID":"4b3084e8-6418-4139-9623-5528c6b2cef3","Type":"ContainerDied","Data":"d497bcc3a800f153d4923e8482db8de3f696d7845d28e3a9a6ddb79278dc8c94"}
Sep 09 03:13:39.635159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:39.635130    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h6lb8" event={"ID":"90fe6e96-2fae-4cc7-97fb-1d46727849ed","Type":"ContainerStarted","Data":"7b592125ae1cee31a45b281f0f7a33454b334782cc5877610c98e1c6f51e6cc7"}
Sep 09 03:13:40.641552 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:40.641354    2560 generic.go:356] "Generic (PLEG): container finished" podID="4b3084e8-6418-4139-9623-5528c6b2cef3" containerID="8b3ee0cdb51dcdc553d2c4447a58684c5930f737966228cd2563405efeeba341" exitCode=0
Sep 09 03:13:40.642095 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:40.641558    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sbbhj" event={"ID":"4b3084e8-6418-4139-9623-5528c6b2cef3","Type":"ContainerDied","Data":"8b3ee0cdb51dcdc553d2c4447a58684c5930f737966228cd2563405efeeba341"}
Sep 09 03:13:40.976092 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:40.975971    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5jzgn\" (UID: \"6b8b2e70-6c05-4bfb-98ce-e18701db3e63\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:13:40.976245 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:40.976106    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:40.976245 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:40.976160    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:13:40.976332 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:40.976246    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert podName:6b8b2e70-6c05-4bfb-98ce-e18701db3e63 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:44.97622461 +0000 UTC m=+42.112972476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5jzgn" (UID: "6b8b2e70-6c05-4bfb-98ce-e18701db3e63") : secret "networking-console-plugin-cert" not found
Sep 09 03:13:40.976332 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:40.976251    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:13:40.976332 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:40.976267    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-94bb5b9d4-65f4d: secret "image-registry-tls" not found
Sep 09 03:13:40.976332 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:40.976306    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls podName:87fed466-b71f-4bc5-b0e0-447ebecbaacb nodeName:}" failed. No retries permitted until 2026-09-09 03:13:44.976293342 +0000 UTC m=+42.113041185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls") pod "image-registry-94bb5b9d4-65f4d" (UID: "87fed466-b71f-4bc5-b0e0-447ebecbaacb") : secret "image-registry-tls" not found
Sep 09 03:13:41.076895 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:41.076860    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:13:41.077057 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:41.077018    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:13:41.077101 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:41.077095    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls podName:a65c3985-522f-46f4-aeb2-7f12ae347816 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:45.077074307 +0000 UTC m=+42.213822163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls") pod "dns-default-c465c" (UID: "a65c3985-522f-46f4-aeb2-7f12ae347816") : secret "dns-default-metrics-tls" not found
Sep 09 03:13:41.647012 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:41.646979    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sbbhj" event={"ID":"4b3084e8-6418-4139-9623-5528c6b2cef3","Type":"ContainerStarted","Data":"9b0eb0e384158a7a3ef65f51dc6d40caa0c6aaa93997f05eb78a2dbf66c840cb"}
Sep 09 03:13:41.648480 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:41.648444    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h6lb8" event={"ID":"90fe6e96-2fae-4cc7-97fb-1d46727849ed","Type":"ContainerStarted","Data":"1f29f0951c571981a4c8c89fb4238ab37aadc9e7be667dc31e71da84fd5a1533"}
Sep 09 03:13:41.667363 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:41.667316    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sbbhj" podStartSLOduration=8.217160532 podStartE2EDuration="37.667300602s" totalImagesPullingTime="29.45014007s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:13:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:41.665732776 +0000 UTC m=+38.802480640" watchObservedRunningTime="2026-09-09 03:13:41.667300602 +0000 UTC m=+38.804048466"
Sep 09 03:13:41.681291 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:41.681244    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h6lb8" podStartSLOduration=2.004000632 podStartE2EDuration="4.681234215s" totalImagesPullingTime="2.677233583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:13:41.680231385 +0000 UTC m=+38.816979249" watchObservedRunningTime="2026-09-09 03:13:41.681234215 +0000 UTC m=+38.817982080"
Sep 09 03:13:41.823093 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:41.823055    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:41.913799 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:41.913725    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:42.757402 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:42.757359    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:42.812201 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:42.812169    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:44.376254 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:44.376224    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:44.429185 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:44.429144    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:45.004201 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:45.004149    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5jzgn\" (UID: \"6b8b2e70-6c05-4bfb-98ce-e18701db3e63\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:13:45.004407 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:45.004231    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:45.004407 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:45.004342    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:13:45.004523 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:45.004412    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert podName:6b8b2e70-6c05-4bfb-98ce-e18701db3e63 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:53.004395915 +0000 UTC m=+50.141143757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5jzgn" (UID: "6b8b2e70-6c05-4bfb-98ce-e18701db3e63") : secret "networking-console-plugin-cert" not found
Sep 09 03:13:45.004523 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:45.004347    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:13:45.004523 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:45.004461    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-94bb5b9d4-65f4d: secret "image-registry-tls" not found
Sep 09 03:13:45.004679 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:45.004523    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls podName:87fed466-b71f-4bc5-b0e0-447ebecbaacb nodeName:}" failed. No retries permitted until 2026-09-09 03:13:53.004505293 +0000 UTC m=+50.141253139 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls") pod "image-registry-94bb5b9d4-65f4d" (UID: "87fed466-b71f-4bc5-b0e0-447ebecbaacb") : secret "image-registry-tls" not found
Sep 09 03:13:45.105211 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:45.105181    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:13:45.105358 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:45.105337    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:13:45.105436 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:45.105415    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls podName:a65c3985-522f-46f4-aeb2-7f12ae347816 nodeName:}" failed. No retries permitted until 2026-09-09 03:13:53.105394763 +0000 UTC m=+50.242142621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls") pod "dns-default-c465c" (UID: "a65c3985-522f-46f4-aeb2-7f12ae347816") : secret "dns-default-metrics-tls" not found
Sep 09 03:13:48.121314 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:48.121282    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:48.178870 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:48.178836    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:48.948249 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:48.948219    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:49.005234 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:49.005205    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:49.924451 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:49.924394    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:49.975973 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:49.975935    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:13:53.066755 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:53.066705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:13:53.067294 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:53.066779    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5jzgn\" (UID: \"6b8b2e70-6c05-4bfb-98ce-e18701db3e63\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:13:53.067294 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:53.066892    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:13:53.067294 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:53.066906    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:13:53.067294 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:53.066916    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-94bb5b9d4-65f4d: secret "image-registry-tls" not found
Sep 09 03:13:53.067294 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:53.066979    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls podName:87fed466-b71f-4bc5-b0e0-447ebecbaacb nodeName:}" failed. No retries permitted until 2026-09-09 03:14:09.066962371 +0000 UTC m=+66.203710230 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls") pod "image-registry-94bb5b9d4-65f4d" (UID: "87fed466-b71f-4bc5-b0e0-447ebecbaacb") : secret "image-registry-tls" not found
Sep 09 03:13:53.067294 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:53.066993    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert podName:6b8b2e70-6c05-4bfb-98ce-e18701db3e63 nodeName:}" failed. No retries permitted until 2026-09-09 03:14:09.066987563 +0000 UTC m=+66.203735406 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5jzgn" (UID: "6b8b2e70-6c05-4bfb-98ce-e18701db3e63") : secret "networking-console-plugin-cert" not found
Sep 09 03:13:53.167343 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:13:53.167305    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:13:53.167513 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:53.167441    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:13:53.167513 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:13:53.167500    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls podName:a65c3985-522f-46f4-aeb2-7f12ae347816 nodeName:}" failed. No retries permitted until 2026-09-09 03:14:09.167485945 +0000 UTC m=+66.304233793 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls") pod "dns-default-c465c" (UID: "a65c3985-522f-46f4-aeb2-7f12ae347816") : secret "dns-default-metrics-tls" not found
Sep 09 03:14:03.626683 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:03.626651    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cjmpt"
Sep 09 03:14:03.775079 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:03.775039    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:03.831106 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:03.831068    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:05.745471 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:05.745436    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:05.799516 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:05.799480    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:09.081294 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.081256    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:14:09.081832 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.081350    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5jzgn\" (UID: \"6b8b2e70-6c05-4bfb-98ce-e18701db3e63\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:14:09.081832 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.081391    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret\") pod \"global-pull-secret-syncer-v89sv\" (UID: \"422df607-4fd9-4edd-ad99-7b6214c7d970\") " pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:14:09.081832 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:14:09.081460    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:14:09.081832 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:14:09.081484    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-94bb5b9d4-65f4d: secret "image-registry-tls" not found
Sep 09 03:14:09.081832 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.081509    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:09.081832 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:14:09.081516    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:14:09.081832 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:14:09.081554    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls podName:87fed466-b71f-4bc5-b0e0-447ebecbaacb nodeName:}" failed. No retries permitted until 2026-09-09 03:14:41.081532553 +0000 UTC m=+98.218280412 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls") pod "image-registry-94bb5b9d4-65f4d" (UID: "87fed466-b71f-4bc5-b0e0-447ebecbaacb") : secret "image-registry-tls" not found
Sep 09 03:14:09.081832 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:14:09.081585    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert podName:6b8b2e70-6c05-4bfb-98ce-e18701db3e63 nodeName:}" failed. No retries permitted until 2026-09-09 03:14:41.081571198 +0000 UTC m=+98.218319043 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5jzgn" (UID: "6b8b2e70-6c05-4bfb-98ce-e18701db3e63") : secret "networking-console-plugin-cert" not found
Sep 09 03:14:09.084053 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.084037    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 03:14:09.095297 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.095277    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/422df607-4fd9-4edd-ad99-7b6214c7d970-original-pull-secret\") pod \"global-pull-secret-syncer-v89sv\" (UID: \"422df607-4fd9-4edd-ad99-7b6214c7d970\") " pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:14:09.181769 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.181742    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:14:09.181889 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.181772    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs\") pod \"network-metrics-daemon-hlmbb\" (UID: \"f8508862-2ffb-4bd8-bb4f-67cd69a78075\") " pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:14:09.181943 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:14:09.181893    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:14:09.182091 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:14:09.181950    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls podName:a65c3985-522f-46f4-aeb2-7f12ae347816 nodeName:}" failed. No retries permitted until 2026-09-09 03:14:41.181930627 +0000 UTC m=+98.318678483 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls") pod "dns-default-c465c" (UID: "a65c3985-522f-46f4-aeb2-7f12ae347816") : secret "dns-default-metrics-tls" not found
Sep 09 03:14:09.182413 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.182392    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:09.184268 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.184252    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 03:14:09.192634 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:14:09.192617    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 03:14:09.192708 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:14:09.192671    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs podName:f8508862-2ffb-4bd8-bb4f-67cd69a78075 nodeName:}" failed. No retries permitted until 2026-09-09 03:15:13.192659635 +0000 UTC m=+130.329407479 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs") pod "network-metrics-daemon-hlmbb" (UID: "f8508862-2ffb-4bd8-bb4f-67cd69a78075") : secret "metrics-daemon-secret" not found
Sep 09 03:14:09.275403 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.275376    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v89sv"
Sep 09 03:14:09.282076 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.282047    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl5z2\" (UniqueName: \"kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2\") pod \"network-check-target-tls5w\" (UID: \"d539727e-a1d4-43f5-be9e-84c6611912be\") " pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:14:09.282460 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.282442    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:09.284720 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.284706    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 03:14:09.294655 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.294638    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 03:14:09.305146 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.305128    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl5z2\" (UniqueName: \"kubernetes.io/projected/d539727e-a1d4-43f5-be9e-84c6611912be-kube-api-access-jl5z2\") pod \"network-check-target-tls5w\" (UID: \"d539727e-a1d4-43f5-be9e-84c6611912be\") " pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:14:09.571501 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.571456    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v89sv"]
Sep 09 03:14:09.573042 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.573020    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g4cct\""
Sep 09 03:14:09.574386 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:14:09.574357    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422df607_4fd9_4edd_ad99_7b6214c7d970.slice/crio-aff6b1c7b7a78c15d71b471767c5263aae376c1d3796364f737f09747bf6ab57 WatchSource:0}: Error finding container aff6b1c7b7a78c15d71b471767c5263aae376c1d3796364f737f09747bf6ab57: Status 404 returned error can't find the container with id aff6b1c7b7a78c15d71b471767c5263aae376c1d3796364f737f09747bf6ab57
Sep 09 03:14:09.581184 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.581166    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:14:09.712434 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.712402    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v89sv" event={"ID":"422df607-4fd9-4edd-ad99-7b6214c7d970","Type":"ContainerStarted","Data":"aff6b1c7b7a78c15d71b471767c5263aae376c1d3796364f737f09747bf6ab57"}
Sep 09 03:14:09.875078 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:09.875056    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tls5w"]
Sep 09 03:14:09.877290 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:14:09.877259    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd539727e_a1d4_43f5_be9e_84c6611912be.slice/crio-d83e110284c772aedc15012bd92ce067be3129c4f916a0cbd72e8342ee0e1213 WatchSource:0}: Error finding container d83e110284c772aedc15012bd92ce067be3129c4f916a0cbd72e8342ee0e1213: Status 404 returned error can't find the container with id d83e110284c772aedc15012bd92ce067be3129c4f916a0cbd72e8342ee0e1213
Sep 09 03:14:10.716867 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:10.716822    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tls5w" event={"ID":"d539727e-a1d4-43f5-be9e-84c6611912be","Type":"ContainerStarted","Data":"d83e110284c772aedc15012bd92ce067be3129c4f916a0cbd72e8342ee0e1213"}
Sep 09 03:14:14.728811 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:14.728722    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v89sv" event={"ID":"422df607-4fd9-4edd-ad99-7b6214c7d970","Type":"ContainerStarted","Data":"3a496bc882287760a8893949bcfc19f00e88aedc2323cfaa4f2c88b36cdab601"}
Sep 09 03:14:14.730370 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:14.730343    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tls5w" event={"ID":"d539727e-a1d4-43f5-be9e-84c6611912be","Type":"ContainerStarted","Data":"0c817910f0b9c72e5b13bb8efc579d75fe75c42b0dc8f03d4fff8a388ccc804c"}
Sep 09 03:14:14.730501 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:14.730480    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:14:14.743074 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:14.743027    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v89sv" podStartSLOduration=65.933970793 podStartE2EDuration="1m10.743016461s" totalImagesPullingTime="4.809045668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:14:14.742497062 +0000 UTC m=+71.879244939" watchObservedRunningTime="2026-09-09 03:14:14.743016461 +0000 UTC m=+71.879764325"
Sep 09 03:14:14.756441 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:14.756410    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tls5w" podStartSLOduration=66.256204276 podStartE2EDuration="1m10.756398764s" totalImagesPullingTime="4.500194488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:14:14.756089774 +0000 UTC m=+71.892837640" watchObservedRunningTime="2026-09-09 03:14:14.756398764 +0000 UTC m=+71.893146607"
Sep 09 03:14:17.357959 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:17.357925    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:17.412717 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:17.412681    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:41.110889 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:41.110740    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5jzgn\" (UID: \"6b8b2e70-6c05-4bfb-98ce-e18701db3e63\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:14:41.110889 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:41.110810    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:14:41.111331 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:14:41.110883    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:14:41.111331 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:14:41.110914    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:14:41.111331 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:14:41.110927    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-94bb5b9d4-65f4d: secret "image-registry-tls" not found
Sep 09 03:14:41.111331 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:14:41.110961    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert podName:6b8b2e70-6c05-4bfb-98ce-e18701db3e63 nodeName:}" failed. No retries permitted until 2026-09-09 03:15:45.11094422 +0000 UTC m=+162.247692067 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5jzgn" (UID: "6b8b2e70-6c05-4bfb-98ce-e18701db3e63") : secret "networking-console-plugin-cert" not found
Sep 09 03:14:41.111331 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:14:41.110987    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls podName:87fed466-b71f-4bc5-b0e0-447ebecbaacb nodeName:}" failed. No retries permitted until 2026-09-09 03:15:45.110966745 +0000 UTC m=+162.247714593 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls") pod "image-registry-94bb5b9d4-65f4d" (UID: "87fed466-b71f-4bc5-b0e0-447ebecbaacb") : secret "image-registry-tls" not found
Sep 09 03:14:41.212009 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:41.211979    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:14:41.212127 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:14:41.212100    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:14:41.212173 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:14:41.212147    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls podName:a65c3985-522f-46f4-aeb2-7f12ae347816 nodeName:}" failed. No retries permitted until 2026-09-09 03:15:45.21213483 +0000 UTC m=+162.348882673 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls") pod "dns-default-c465c" (UID: "a65c3985-522f-46f4-aeb2-7f12ae347816") : secret "dns-default-metrics-tls" not found
Sep 09 03:14:43.977374 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:43.977335    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:44.025102 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:44.025073    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:14:45.735976 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:14:45.735947    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tls5w"
Sep 09 03:15:13.242492 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:15:13.242428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs\") pod \"network-metrics-daemon-hlmbb\" (UID: \"f8508862-2ffb-4bd8-bb4f-67cd69a78075\") " pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:15:13.242955 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:15:13.242585    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 03:15:13.242955 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:15:13.242677    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs podName:f8508862-2ffb-4bd8-bb4f-67cd69a78075 nodeName:}" failed. No retries permitted until 2026-09-09 03:17:15.242660141 +0000 UTC m=+252.379407984 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs") pod "network-metrics-daemon-hlmbb" (UID: "f8508862-2ffb-4bd8-bb4f-67cd69a78075") : secret "metrics-daemon-secret" not found
Sep 09 03:15:40.237123 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:15:40.237081    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn" podUID="6b8b2e70-6c05-4bfb-98ce-e18701db3e63"
Sep 09 03:15:40.251267 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:15:40.251242    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d" podUID="87fed466-b71f-4bc5-b0e0-447ebecbaacb"
Sep 09 03:15:40.293579 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:15:40.293539    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-c465c" podUID="a65c3985-522f-46f4-aeb2-7f12ae347816"
Sep 09 03:15:40.936151 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:15:40.936122    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:15:40.936313 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:15:40.936122    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c465c"
Sep 09 03:15:40.936313 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:15:40.936126    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:15:41.362895 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:15:41.362856    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:15:45.156425 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:15:45.156362    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:15:45.156874 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:15:45.156447    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5jzgn\" (UID: \"6b8b2e70-6c05-4bfb-98ce-e18701db3e63\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:15:45.156874 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:15:45.156498    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:15:45.156874 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:15:45.156520    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-94bb5b9d4-65f4d: secret "image-registry-tls" not found
Sep 09 03:15:45.156874 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:15:45.156547    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:15:45.156874 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:15:45.156587    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls podName:87fed466-b71f-4bc5-b0e0-447ebecbaacb nodeName:}" failed. No retries permitted until 2026-09-09 03:17:47.156569354 +0000 UTC m=+284.293317197 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls") pod "image-registry-94bb5b9d4-65f4d" (UID: "87fed466-b71f-4bc5-b0e0-447ebecbaacb") : secret "image-registry-tls" not found
Sep 09 03:15:45.156874 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:15:45.156619    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert podName:6b8b2e70-6c05-4bfb-98ce-e18701db3e63 nodeName:}" failed. No retries permitted until 2026-09-09 03:17:47.156612671 +0000 UTC m=+284.293360514 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5jzgn" (UID: "6b8b2e70-6c05-4bfb-98ce-e18701db3e63") : secret "networking-console-plugin-cert" not found
Sep 09 03:15:45.256726 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:15:45.256690    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:15:45.256872 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:15:45.256801    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:15:45.256872 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:15:45.256862    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls podName:a65c3985-522f-46f4-aeb2-7f12ae347816 nodeName:}" failed. No retries permitted until 2026-09-09 03:17:47.256847495 +0000 UTC m=+284.393595338 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls") pod "dns-default-c465c" (UID: "a65c3985-522f-46f4-aeb2-7f12ae347816") : secret "dns-default-metrics-tls" not found
Sep 09 03:15:52.353293 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:15:52.353257    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:16:30.898672 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:16:30.898638    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:54.779366 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:16:54.779331    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:16:57.323364 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:16:57.323333    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:15.294118 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:15.294079    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs\") pod \"network-metrics-daemon-hlmbb\" (UID: \"f8508862-2ffb-4bd8-bb4f-67cd69a78075\") " pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:17:15.294578 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:15.294214    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 03:17:15.294578 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:15.294293    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs podName:f8508862-2ffb-4bd8-bb4f-67cd69a78075 nodeName:}" failed. No retries permitted until 2026-09-09 03:19:17.294276851 +0000 UTC m=+374.431024695 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs") pod "network-metrics-daemon-hlmbb" (UID: "f8508862-2ffb-4bd8-bb4f-67cd69a78075") : secret "metrics-daemon-secret" not found
Sep 09 03:17:23.641712 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.641679    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7x7k4"]
Sep 09 03:17:23.644400 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.644384    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7x7k4"
Sep 09 03:17:23.646600 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.646573    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-vhcbz"]
Sep 09 03:17:23.646893 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.646868    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ngs2\" (UniqueName: \"kubernetes.io/projected/fa341cd0-a17e-4677-978d-3a3abcde5ad3-kube-api-access-6ngs2\") pod \"volume-data-source-validator-695c895bd6-7x7k4\" (UID: \"fa341cd0-a17e-4677-978d-3a3abcde5ad3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7x7k4"
Sep 09 03:17:23.647020 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.646999    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:17:23.647101 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.647029    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2h57k\""
Sep 09 03:17:23.647101 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.647077    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 09 03:17:23.649338 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.649317    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-jtgvc"]
Sep 09 03:17:23.649440 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.649420    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-vhcbz"
Sep 09 03:17:23.652046 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.652026    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"]
Sep 09 03:17:23.652156 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.652058    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 03:17:23.652156 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.652073    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 03:17:23.652156 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.652133    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-vw848\""
Sep 09 03:17:23.652329 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.652218    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.652677 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.652660    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 03:17:23.652778 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.652735    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:17:23.654912 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.654893    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"
Sep 09 03:17:23.655088 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.655065    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7x7k4"]
Sep 09 03:17:23.656249 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.656229    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-c2hcw\""
Sep 09 03:17:23.657092 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.657049    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 03:17:23.657569 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.657551    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 03:17:23.657675 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.657575    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 03:17:23.657733 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.657705    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 03:17:23.657811 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.657784    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 03:17:23.657928 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.657808    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 03:17:23.659258 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.659224    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 03:17:23.660726 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.660707    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9d4d4\""
Sep 09 03:17:23.660867 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.660841    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 03:17:23.662377 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.662357    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 03:17:23.663667 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.663643    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-vhcbz"]
Sep 09 03:17:23.665508 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.665490    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-jtgvc"]
Sep 09 03:17:23.665796 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.665774    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 03:17:23.675431 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.675411    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"]
Sep 09 03:17:23.740239 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.740207    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn"]
Sep 09 03:17:23.743131 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.743116    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn"
Sep 09 03:17:23.745553 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.745530    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 09 03:17:23.745722 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.745562    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 09 03:17:23.745722 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.745620    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 09 03:17:23.745722 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.745562    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:17:23.745901 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.745827    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ld6jl\""
Sep 09 03:17:23.747509 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.747489    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d8945e0-9f39-4523-b066-5125ad017318-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dhtsn\" (UID: \"9d8945e0-9f39-4523-b066-5125ad017318\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn"
Sep 09 03:17:23.747606 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.747529    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ngs2\" (UniqueName: \"kubernetes.io/projected/fa341cd0-a17e-4677-978d-3a3abcde5ad3-kube-api-access-6ngs2\") pod \"volume-data-source-validator-695c895bd6-7x7k4\" (UID: \"fa341cd0-a17e-4677-978d-3a3abcde5ad3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7x7k4"
Sep 09 03:17:23.747606 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.747556    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8945e0-9f39-4523-b066-5125ad017318-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dhtsn\" (UID: \"9d8945e0-9f39-4523-b066-5125ad017318\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn"
Sep 09 03:17:23.747684 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.747620    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-jtgvc\" (UID: \"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.747684 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.747652    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9dlq\" (UniqueName: \"kubernetes.io/projected/9d8945e0-9f39-4523-b066-5125ad017318-kube-api-access-v9dlq\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dhtsn\" (UID: \"9d8945e0-9f39-4523-b066-5125ad017318\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn"
Sep 09 03:17:23.747787 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.747697    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593-serving-cert\") pod \"insights-operator-76bcd6ccf5-jtgvc\" (UID: \"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.747787 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.747754    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ebf6c83-e589-4833-994e-25ebfbb0b0a1-serving-cert\") pod \"console-operator-85b454874b-vhcbz\" (UID: \"5ebf6c83-e589-4833-994e-25ebfbb0b0a1\") " pod="openshift-console-operator/console-operator-85b454874b-vhcbz"
Sep 09 03:17:23.747787 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.747779    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-snfbl\" (UID: \"aec25042-0f8f-4bda-a8bf-f1adb30990a6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"
Sep 09 03:17:23.747915 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.747806    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94mdj\" (UniqueName: \"kubernetes.io/projected/5ebf6c83-e589-4833-994e-25ebfbb0b0a1-kube-api-access-94mdj\") pod \"console-operator-85b454874b-vhcbz\" (UID: \"5ebf6c83-e589-4833-994e-25ebfbb0b0a1\") " pod="openshift-console-operator/console-operator-85b454874b-vhcbz"
Sep 09 03:17:23.747915 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.747832    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ebf6c83-e589-4833-994e-25ebfbb0b0a1-config\") pod \"console-operator-85b454874b-vhcbz\" (UID: \"5ebf6c83-e589-4833-994e-25ebfbb0b0a1\") " pod="openshift-console-operator/console-operator-85b454874b-vhcbz"
Sep 09 03:17:23.747915 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.747852    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/aec25042-0f8f-4bda-a8bf-f1adb30990a6-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-snfbl\" (UID: \"aec25042-0f8f-4bda-a8bf-f1adb30990a6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"
Sep 09 03:17:23.747915 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.747867    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593-tmp\") pod \"insights-operator-76bcd6ccf5-jtgvc\" (UID: \"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.748058 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.747937    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-jtgvc\" (UID: \"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.748058 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.747976    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ebf6c83-e589-4833-994e-25ebfbb0b0a1-trusted-ca\") pod \"console-operator-85b454874b-vhcbz\" (UID: \"5ebf6c83-e589-4833-994e-25ebfbb0b0a1\") " pod="openshift-console-operator/console-operator-85b454874b-vhcbz"
Sep 09 03:17:23.748058 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.747998    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593-snapshots\") pod \"insights-operator-76bcd6ccf5-jtgvc\" (UID: \"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.748058 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.748017    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l84cx\" (UniqueName: \"kubernetes.io/projected/6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593-kube-api-access-l84cx\") pod \"insights-operator-76bcd6ccf5-jtgvc\" (UID: \"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.748303 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.748061    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-286fx\" (UniqueName: \"kubernetes.io/projected/aec25042-0f8f-4bda-a8bf-f1adb30990a6-kube-api-access-286fx\") pod \"cluster-monitoring-operator-6cdcc96965-snfbl\" (UID: \"aec25042-0f8f-4bda-a8bf-f1adb30990a6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"
Sep 09 03:17:23.755681 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.755652    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn"]
Sep 09 03:17:23.757287 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.757263    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ngs2\" (UniqueName: \"kubernetes.io/projected/fa341cd0-a17e-4677-978d-3a3abcde5ad3-kube-api-access-6ngs2\") pod \"volume-data-source-validator-695c895bd6-7x7k4\" (UID: \"fa341cd0-a17e-4677-978d-3a3abcde5ad3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7x7k4"
Sep 09 03:17:23.848414 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.848376    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-286fx\" (UniqueName: \"kubernetes.io/projected/aec25042-0f8f-4bda-a8bf-f1adb30990a6-kube-api-access-286fx\") pod \"cluster-monitoring-operator-6cdcc96965-snfbl\" (UID: \"aec25042-0f8f-4bda-a8bf-f1adb30990a6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"
Sep 09 03:17:23.848579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.848425    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d8945e0-9f39-4523-b066-5125ad017318-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dhtsn\" (UID: \"9d8945e0-9f39-4523-b066-5125ad017318\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn"
Sep 09 03:17:23.848579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.848532    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8945e0-9f39-4523-b066-5125ad017318-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dhtsn\" (UID: \"9d8945e0-9f39-4523-b066-5125ad017318\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn"
Sep 09 03:17:23.848579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.848563    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-jtgvc\" (UID: \"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.848789 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.848616    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9dlq\" (UniqueName: \"kubernetes.io/projected/9d8945e0-9f39-4523-b066-5125ad017318-kube-api-access-v9dlq\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dhtsn\" (UID: \"9d8945e0-9f39-4523-b066-5125ad017318\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn"
Sep 09 03:17:23.848789 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.848647    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593-serving-cert\") pod \"insights-operator-76bcd6ccf5-jtgvc\" (UID: \"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.848789 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.848702    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ebf6c83-e589-4833-994e-25ebfbb0b0a1-serving-cert\") pod \"console-operator-85b454874b-vhcbz\" (UID: \"5ebf6c83-e589-4833-994e-25ebfbb0b0a1\") " pod="openshift-console-operator/console-operator-85b454874b-vhcbz"
Sep 09 03:17:23.848789 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.848727    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-snfbl\" (UID: \"aec25042-0f8f-4bda-a8bf-f1adb30990a6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"
Sep 09 03:17:23.848961 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.848944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94mdj\" (UniqueName: \"kubernetes.io/projected/5ebf6c83-e589-4833-994e-25ebfbb0b0a1-kube-api-access-94mdj\") pod \"console-operator-85b454874b-vhcbz\" (UID: \"5ebf6c83-e589-4833-994e-25ebfbb0b0a1\") " pod="openshift-console-operator/console-operator-85b454874b-vhcbz"
Sep 09 03:17:23.849010 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.848978    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ebf6c83-e589-4833-994e-25ebfbb0b0a1-config\") pod \"console-operator-85b454874b-vhcbz\" (UID: \"5ebf6c83-e589-4833-994e-25ebfbb0b0a1\") " pod="openshift-console-operator/console-operator-85b454874b-vhcbz"
Sep 09 03:17:23.849063 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.849006    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/aec25042-0f8f-4bda-a8bf-f1adb30990a6-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-snfbl\" (UID: \"aec25042-0f8f-4bda-a8bf-f1adb30990a6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"
Sep 09 03:17:23.849063 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.849030    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593-tmp\") pod \"insights-operator-76bcd6ccf5-jtgvc\" (UID: \"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.849165 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.849054    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d8945e0-9f39-4523-b066-5125ad017318-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dhtsn\" (UID: \"9d8945e0-9f39-4523-b066-5125ad017318\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn"
Sep 09 03:17:23.849165 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.849146    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-jtgvc\" (UID: \"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.849267 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.849173    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ebf6c83-e589-4833-994e-25ebfbb0b0a1-trusted-ca\") pod \"console-operator-85b454874b-vhcbz\" (UID: \"5ebf6c83-e589-4833-994e-25ebfbb0b0a1\") " pod="openshift-console-operator/console-operator-85b454874b-vhcbz"
Sep 09 03:17:23.849267 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.849197    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593-snapshots\") pod \"insights-operator-76bcd6ccf5-jtgvc\" (UID: \"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.849267 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.849207    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-jtgvc\" (UID: \"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.849267 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.849227    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l84cx\" (UniqueName: \"kubernetes.io/projected/6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593-kube-api-access-l84cx\") pod \"insights-operator-76bcd6ccf5-jtgvc\" (UID: \"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.849506 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:23.849489    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 03:17:23.849664 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:23.849650    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls podName:aec25042-0f8f-4bda-a8bf-f1adb30990a6 nodeName:}" failed. No retries permitted until 2026-09-09 03:17:24.349629536 +0000 UTC m=+261.486377420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-snfbl" (UID: "aec25042-0f8f-4bda-a8bf-f1adb30990a6") : secret "cluster-monitoring-operator-tls" not found
Sep 09 03:17:23.849794 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.849770    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ebf6c83-e589-4833-994e-25ebfbb0b0a1-config\") pod \"console-operator-85b454874b-vhcbz\" (UID: \"5ebf6c83-e589-4833-994e-25ebfbb0b0a1\") " pod="openshift-console-operator/console-operator-85b454874b-vhcbz"
Sep 09 03:17:23.849940 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.849844    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/aec25042-0f8f-4bda-a8bf-f1adb30990a6-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-snfbl\" (UID: \"aec25042-0f8f-4bda-a8bf-f1adb30990a6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"
Sep 09 03:17:23.850009 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.849962    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ebf6c83-e589-4833-994e-25ebfbb0b0a1-trusted-ca\") pod \"console-operator-85b454874b-vhcbz\" (UID: \"5ebf6c83-e589-4833-994e-25ebfbb0b0a1\") " pod="openshift-console-operator/console-operator-85b454874b-vhcbz"
Sep 09 03:17:23.850065 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.850022    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593-tmp\") pod \"insights-operator-76bcd6ccf5-jtgvc\" (UID: \"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.850119 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.850088    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593-snapshots\") pod \"insights-operator-76bcd6ccf5-jtgvc\" (UID: \"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.850327 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.850301    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-jtgvc\" (UID: \"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.851364 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.851340    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8945e0-9f39-4523-b066-5125ad017318-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dhtsn\" (UID: \"9d8945e0-9f39-4523-b066-5125ad017318\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn"
Sep 09 03:17:23.852558 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.852531    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ebf6c83-e589-4833-994e-25ebfbb0b0a1-serving-cert\") pod \"console-operator-85b454874b-vhcbz\" (UID: \"5ebf6c83-e589-4833-994e-25ebfbb0b0a1\") " pod="openshift-console-operator/console-operator-85b454874b-vhcbz"
Sep 09 03:17:23.852667 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.852643    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593-serving-cert\") pod \"insights-operator-76bcd6ccf5-jtgvc\" (UID: \"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.856955 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.856930    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9dlq\" (UniqueName: \"kubernetes.io/projected/9d8945e0-9f39-4523-b066-5125ad017318-kube-api-access-v9dlq\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dhtsn\" (UID: \"9d8945e0-9f39-4523-b066-5125ad017318\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn"
Sep 09 03:17:23.857058 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.857020    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-286fx\" (UniqueName: \"kubernetes.io/projected/aec25042-0f8f-4bda-a8bf-f1adb30990a6-kube-api-access-286fx\") pod \"cluster-monitoring-operator-6cdcc96965-snfbl\" (UID: \"aec25042-0f8f-4bda-a8bf-f1adb30990a6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"
Sep 09 03:17:23.857474 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.857455    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94mdj\" (UniqueName: \"kubernetes.io/projected/5ebf6c83-e589-4833-994e-25ebfbb0b0a1-kube-api-access-94mdj\") pod \"console-operator-85b454874b-vhcbz\" (UID: \"5ebf6c83-e589-4833-994e-25ebfbb0b0a1\") " pod="openshift-console-operator/console-operator-85b454874b-vhcbz"
Sep 09 03:17:23.858424 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.858385    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l84cx\" (UniqueName: \"kubernetes.io/projected/6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593-kube-api-access-l84cx\") pod \"insights-operator-76bcd6ccf5-jtgvc\" (UID: \"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.888048 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.888020    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:23.954869 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.954799    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7x7k4"
Sep 09 03:17:23.962925 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.962902    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-vhcbz"
Sep 09 03:17:23.969697 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.969670    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc"
Sep 09 03:17:23.988036 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:23.988004    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:24.054997 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:24.054972    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn"
Sep 09 03:17:24.228108 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:24.228033    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7x7k4"]
Sep 09 03:17:24.231046 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:17:24.231022    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa341cd0_a17e_4677_978d_3a3abcde5ad3.slice/crio-bc2d3f64e130bdcc6239f5d1e52092bc49938c3e37a71f0035a38a6bd6acc8be WatchSource:0}: Error finding container bc2d3f64e130bdcc6239f5d1e52092bc49938c3e37a71f0035a38a6bd6acc8be: Status 404 returned error can't find the container with id bc2d3f64e130bdcc6239f5d1e52092bc49938c3e37a71f0035a38a6bd6acc8be
Sep 09 03:17:24.234191 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:24.234170    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-vhcbz"]
Sep 09 03:17:24.235581 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:17:24.235559    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ebf6c83_e589_4833_994e_25ebfbb0b0a1.slice/crio-90daa6a485a21dab8beac6c30cd3ea15a8e36fd0ae08835920a5a557bf59ae5e WatchSource:0}: Error finding container 90daa6a485a21dab8beac6c30cd3ea15a8e36fd0ae08835920a5a557bf59ae5e: Status 404 returned error can't find the container with id 90daa6a485a21dab8beac6c30cd3ea15a8e36fd0ae08835920a5a557bf59ae5e
Sep 09 03:17:24.254267 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:24.254242    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-jtgvc"]
Sep 09 03:17:24.256644 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:17:24.256623    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fea3e4f_7ff0_4f2d_8d8f_9dfda7ae8593.slice/crio-68dc7f08bdc643f594088b32b33a8c4126a61e6b26fddecd1c70feb44f8498df WatchSource:0}: Error finding container 68dc7f08bdc643f594088b32b33a8c4126a61e6b26fddecd1c70feb44f8498df: Status 404 returned error can't find the container with id 68dc7f08bdc643f594088b32b33a8c4126a61e6b26fddecd1c70feb44f8498df
Sep 09 03:17:24.322117 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:24.322039    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn"]
Sep 09 03:17:24.324472 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:17:24.324445    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8945e0_9f39_4523_b066_5125ad017318.slice/crio-2be95d6e6c8f5f816b0827f881ace6aed4a2bbd977885fe918997b38aa3007a0 WatchSource:0}: Error finding container 2be95d6e6c8f5f816b0827f881ace6aed4a2bbd977885fe918997b38aa3007a0: Status 404 returned error can't find the container with id 2be95d6e6c8f5f816b0827f881ace6aed4a2bbd977885fe918997b38aa3007a0
Sep 09 03:17:24.352845 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:24.352806    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-snfbl\" (UID: \"aec25042-0f8f-4bda-a8bf-f1adb30990a6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"
Sep 09 03:17:24.352998 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:24.352947    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 03:17:24.353037 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:24.353008    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls podName:aec25042-0f8f-4bda-a8bf-f1adb30990a6 nodeName:}" failed. No retries permitted until 2026-09-09 03:17:25.352992957 +0000 UTC m=+262.489740800 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-snfbl" (UID: "aec25042-0f8f-4bda-a8bf-f1adb30990a6") : secret "cluster-monitoring-operator-tls" not found
Sep 09 03:17:25.190929 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:25.190892    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc" event={"ID":"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593","Type":"ContainerStarted","Data":"68dc7f08bdc643f594088b32b33a8c4126a61e6b26fddecd1c70feb44f8498df"}
Sep 09 03:17:25.193921 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:25.193887    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7x7k4" event={"ID":"fa341cd0-a17e-4677-978d-3a3abcde5ad3","Type":"ContainerStarted","Data":"bc2d3f64e130bdcc6239f5d1e52092bc49938c3e37a71f0035a38a6bd6acc8be"}
Sep 09 03:17:25.195877 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:25.195843    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn" event={"ID":"9d8945e0-9f39-4523-b066-5125ad017318","Type":"ContainerStarted","Data":"2be95d6e6c8f5f816b0827f881ace6aed4a2bbd977885fe918997b38aa3007a0"}
Sep 09 03:17:25.197606 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:25.197571    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-vhcbz" event={"ID":"5ebf6c83-e589-4833-994e-25ebfbb0b0a1","Type":"ContainerStarted","Data":"90daa6a485a21dab8beac6c30cd3ea15a8e36fd0ae08835920a5a557bf59ae5e"}
Sep 09 03:17:25.361499 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:25.361459    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-snfbl\" (UID: \"aec25042-0f8f-4bda-a8bf-f1adb30990a6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"
Sep 09 03:17:25.361679 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:25.361658    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 03:17:25.361754 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:25.361742    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls podName:aec25042-0f8f-4bda-a8bf-f1adb30990a6 nodeName:}" failed. No retries permitted until 2026-09-09 03:17:27.361720947 +0000 UTC m=+264.498468810 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-snfbl" (UID: "aec25042-0f8f-4bda-a8bf-f1adb30990a6") : secret "cluster-monitoring-operator-tls" not found
Sep 09 03:17:27.376731 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:27.376701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-snfbl\" (UID: \"aec25042-0f8f-4bda-a8bf-f1adb30990a6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"
Sep 09 03:17:27.377002 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:27.376834    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 03:17:27.377002 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:27.376911    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls podName:aec25042-0f8f-4bda-a8bf-f1adb30990a6 nodeName:}" failed. No retries permitted until 2026-09-09 03:17:31.376892889 +0000 UTC m=+268.513640736 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-snfbl" (UID: "aec25042-0f8f-4bda-a8bf-f1adb30990a6") : secret "cluster-monitoring-operator-tls" not found
Sep 09 03:17:27.458584 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:27.458505    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:27.541434 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:27.541401    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:27.898345 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:27.898313    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:28.032474 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:28.032444    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:28.164582 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:28.164505    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:28.213273 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:28.213231    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn" event={"ID":"9d8945e0-9f39-4523-b066-5125ad017318","Type":"ContainerStarted","Data":"39ffdf2d8d77f4ca6802ff8c4504e1a4c86f4fcc22dbda7f59a5c306b9d418c8"}
Sep 09 03:17:28.214873 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:28.214847    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/0.log"
Sep 09 03:17:28.215000 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:28.214882    2560 generic.go:356] "Generic (PLEG): container finished" podID="5ebf6c83-e589-4833-994e-25ebfbb0b0a1" containerID="a8976ca40ef4b06211ebf3ddf468b998cb47fd995ecf4f1f23b0678f3e033308" exitCode=255
Sep 09 03:17:28.215000 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:28.214909    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-vhcbz" event={"ID":"5ebf6c83-e589-4833-994e-25ebfbb0b0a1","Type":"ContainerDied","Data":"a8976ca40ef4b06211ebf3ddf468b998cb47fd995ecf4f1f23b0678f3e033308"}
Sep 09 03:17:28.215147 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:28.215130    2560 scope.go:117] "RemoveContainer" containerID="a8976ca40ef4b06211ebf3ddf468b998cb47fd995ecf4f1f23b0678f3e033308"
Sep 09 03:17:28.216691 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:28.216671    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc" event={"ID":"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593","Type":"ContainerStarted","Data":"c6976c45417e675b04729bfe31777a8e3d449955cf3eb90c2cd184e5206242bf"}
Sep 09 03:17:28.218304 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:28.218286    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7x7k4" event={"ID":"fa341cd0-a17e-4677-978d-3a3abcde5ad3","Type":"ContainerStarted","Data":"1c339b4d109aba8cdfea06f01b7352467f41e40c8ab45f790f439fe00cc81924"}
Sep 09 03:17:28.234905 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:28.234500    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn" podStartSLOduration=2.342777405 podStartE2EDuration="5.234480629s" totalImagesPullingTime="2.891703224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:17:28.233627063 +0000 UTC m=+265.370374946" watchObservedRunningTime="2026-09-09 03:17:28.234480629 +0000 UTC m=+265.371228494"
Sep 09 03:17:28.239790 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:28.239742    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:28.249642 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:28.249575    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7x7k4" podStartSLOduration=3.80309807 podStartE2EDuration="5.249555455s" totalImagesPullingTime="1.446457385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:17:28.249499544 +0000 UTC m=+265.386247413" watchObservedRunningTime="2026-09-09 03:17:28.249555455 +0000 UTC m=+265.386303320"
Sep 09 03:17:28.285699 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:28.285651    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc" podStartSLOduration=2.327365308 podStartE2EDuration="5.285631617s" totalImagesPullingTime="2.958266309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:17:28.284534447 +0000 UTC m=+265.421282312" watchObservedRunningTime="2026-09-09 03:17:28.285631617 +0000 UTC m=+265.422379499"
Sep 09 03:17:28.329717 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:28.329685    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:28.783111 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:28.783072    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:28.884277 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:28.884232    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:29.093983 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.093911    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-85tbf"]
Sep 09 03:17:29.097044 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.097024    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:29.099421 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.099398    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h5g47\""
Sep 09 03:17:29.099625 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.099585    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 03:17:29.099718 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.099624    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 03:17:29.106868 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.106849    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-85tbf"]
Sep 09 03:17:29.111056 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.111038    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:29.170122 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.170082    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:29.192479 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.192452    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-data-volume\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:29.192637 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.192497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:29.192637 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.192527    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-crio-socket\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:29.192637 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.192547    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp885\" (UniqueName: \"kubernetes.io/projected/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-api-access-lp885\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:29.192637 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.192574    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:29.222223 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.222200    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:17:29.222664 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.222650    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/0.log"
Sep 09 03:17:29.222740 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.222683    2560 generic.go:356] "Generic (PLEG): container finished" podID="5ebf6c83-e589-4833-994e-25ebfbb0b0a1" containerID="14ce0a4405a187f5c4c43d5751486a816111af416519d300343dcb2b9872d0fa" exitCode=255
Sep 09 03:17:29.222740 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.222710    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-vhcbz" event={"ID":"5ebf6c83-e589-4833-994e-25ebfbb0b0a1","Type":"ContainerDied","Data":"14ce0a4405a187f5c4c43d5751486a816111af416519d300343dcb2b9872d0fa"}
Sep 09 03:17:29.223474 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.223457    2560 scope.go:117] "RemoveContainer" containerID="a8976ca40ef4b06211ebf3ddf468b998cb47fd995ecf4f1f23b0678f3e033308"
Sep 09 03:17:29.223566 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.223552    2560 scope.go:117] "RemoveContainer" containerID="14ce0a4405a187f5c4c43d5751486a816111af416519d300343dcb2b9872d0fa"
Sep 09 03:17:29.223720 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:29.223702    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-vhcbz_openshift-console-operator(5ebf6c83-e589-4833-994e-25ebfbb0b0a1)\"" pod="openshift-console-operator/console-operator-85b454874b-vhcbz" podUID="5ebf6c83-e589-4833-994e-25ebfbb0b0a1"
Sep 09 03:17:29.293163 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.293128    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:29.293536 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:29.293519    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:17:29.293654 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:29.293574    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-insights-runtime-extractor-tls podName:264d64b1-2bf7-4fea-acdd-0381a36e2e0c nodeName:}" failed. No retries permitted until 2026-09-09 03:17:29.793560129 +0000 UTC m=+266.930307971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-85tbf" (UID: "264d64b1-2bf7-4fea-acdd-0381a36e2e0c") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:17:29.293793 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.293765    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-data-volume\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:29.294018 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.293992    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:29.294108 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.294037    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-crio-socket\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:29.294108 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.294059    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-data-volume\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:29.294108 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.294065    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp885\" (UniqueName: \"kubernetes.io/projected/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-api-access-lp885\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:29.294108 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:29.294064    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:17:29.294266 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.294115    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-crio-socket\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:29.294266 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:29.294168    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-rbac-proxy-cm podName:264d64b1-2bf7-4fea-acdd-0381a36e2e0c nodeName:}" failed. No retries permitted until 2026-09-09 03:17:29.794155242 +0000 UTC m=+266.930903088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-85tbf" (UID: "264d64b1-2bf7-4fea-acdd-0381a36e2e0c") : configmap "kube-rbac-proxy" not found
Sep 09 03:17:29.305161 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.305142    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp885\" (UniqueName: \"kubernetes.io/projected/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-api-access-lp885\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:29.771437 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.771404    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-4hq44"]
Sep 09 03:17:29.775585 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.775569    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-4hq44"
Sep 09 03:17:29.778570 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.778543    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 03:17:29.778712 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.778695    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 03:17:29.779847 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.779827    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 03:17:29.779847 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.779833    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 03:17:29.780003 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.779833    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xq4cl\""
Sep 09 03:17:29.783821 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.783800    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-4hq44"]
Sep 09 03:17:29.797046 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.797024    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:29.797138 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.797075    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htb4d\" (UniqueName: \"kubernetes.io/projected/928cd3f5-4077-4ab0-bd5f-e191d30bb2ae-kube-api-access-htb4d\") pod \"service-ca-75d99c484-4hq44\" (UID: \"928cd3f5-4077-4ab0-bd5f-e191d30bb2ae\") " pod="openshift-service-ca/service-ca-75d99c484-4hq44"
Sep 09 03:17:29.797138 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:29.797105    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:17:29.797138 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.797129    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:29.797289 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:29.797150    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-rbac-proxy-cm podName:264d64b1-2bf7-4fea-acdd-0381a36e2e0c nodeName:}" failed. No retries permitted until 2026-09-09 03:17:30.797136871 +0000 UTC m=+267.933884713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-85tbf" (UID: "264d64b1-2bf7-4fea-acdd-0381a36e2e0c") : configmap "kube-rbac-proxy" not found
Sep 09 03:17:29.797289 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:29.797215    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:17:29.797289 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.797220    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/928cd3f5-4077-4ab0-bd5f-e191d30bb2ae-signing-cabundle\") pod \"service-ca-75d99c484-4hq44\" (UID: \"928cd3f5-4077-4ab0-bd5f-e191d30bb2ae\") " pod="openshift-service-ca/service-ca-75d99c484-4hq44"
Sep 09 03:17:29.797289 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:29.797264    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-insights-runtime-extractor-tls podName:264d64b1-2bf7-4fea-acdd-0381a36e2e0c nodeName:}" failed. No retries permitted until 2026-09-09 03:17:30.797249557 +0000 UTC m=+267.933997427 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-85tbf" (UID: "264d64b1-2bf7-4fea-acdd-0381a36e2e0c") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:17:29.797503 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.797307    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/928cd3f5-4077-4ab0-bd5f-e191d30bb2ae-signing-key\") pod \"service-ca-75d99c484-4hq44\" (UID: \"928cd3f5-4077-4ab0-bd5f-e191d30bb2ae\") " pod="openshift-service-ca/service-ca-75d99c484-4hq44"
Sep 09 03:17:29.823051 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.823026    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:29.898657 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.898627    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/928cd3f5-4077-4ab0-bd5f-e191d30bb2ae-signing-key\") pod \"service-ca-75d99c484-4hq44\" (UID: \"928cd3f5-4077-4ab0-bd5f-e191d30bb2ae\") " pod="openshift-service-ca/service-ca-75d99c484-4hq44"
Sep 09 03:17:29.898751 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.898692    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htb4d\" (UniqueName: \"kubernetes.io/projected/928cd3f5-4077-4ab0-bd5f-e191d30bb2ae-kube-api-access-htb4d\") pod \"service-ca-75d99c484-4hq44\" (UID: \"928cd3f5-4077-4ab0-bd5f-e191d30bb2ae\") " pod="openshift-service-ca/service-ca-75d99c484-4hq44"
Sep 09 03:17:29.898826 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.898751    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/928cd3f5-4077-4ab0-bd5f-e191d30bb2ae-signing-cabundle\") pod \"service-ca-75d99c484-4hq44\" (UID: \"928cd3f5-4077-4ab0-bd5f-e191d30bb2ae\") " pod="openshift-service-ca/service-ca-75d99c484-4hq44"
Sep 09 03:17:29.899359 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.899336    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/928cd3f5-4077-4ab0-bd5f-e191d30bb2ae-signing-cabundle\") pod \"service-ca-75d99c484-4hq44\" (UID: \"928cd3f5-4077-4ab0-bd5f-e191d30bb2ae\") " pod="openshift-service-ca/service-ca-75d99c484-4hq44"
Sep 09 03:17:29.900983 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.900963    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/928cd3f5-4077-4ab0-bd5f-e191d30bb2ae-signing-key\") pod \"service-ca-75d99c484-4hq44\" (UID: \"928cd3f5-4077-4ab0-bd5f-e191d30bb2ae\") " pod="openshift-service-ca/service-ca-75d99c484-4hq44"
Sep 09 03:17:29.908481 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.908456    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htb4d\" (UniqueName: \"kubernetes.io/projected/928cd3f5-4077-4ab0-bd5f-e191d30bb2ae-kube-api-access-htb4d\") pod \"service-ca-75d99c484-4hq44\" (UID: \"928cd3f5-4077-4ab0-bd5f-e191d30bb2ae\") " pod="openshift-service-ca/service-ca-75d99c484-4hq44"
Sep 09 03:17:29.926471 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:29.926440    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:30.085072 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:30.084994    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-4hq44"
Sep 09 03:17:30.227550 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:30.227523    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:17:30.227867 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:30.227852    2560 scope.go:117] "RemoveContainer" containerID="14ce0a4405a187f5c4c43d5751486a816111af416519d300343dcb2b9872d0fa"
Sep 09 03:17:30.228005 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:30.227987    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-vhcbz_openshift-console-operator(5ebf6c83-e589-4833-994e-25ebfbb0b0a1)\"" pod="openshift-console-operator/console-operator-85b454874b-vhcbz" podUID="5ebf6c83-e589-4833-994e-25ebfbb0b0a1"
Sep 09 03:17:30.356666 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:30.356625    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-4hq44"]
Sep 09 03:17:30.358563 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:17:30.358532    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928cd3f5_4077_4ab0_bd5f_e191d30bb2ae.slice/crio-836754fc36dbe8d820084c1243a9c937f5e4f882a0f2d7149ca255c46aea7a2d WatchSource:0}: Error finding container 836754fc36dbe8d820084c1243a9c937f5e4f882a0f2d7149ca255c46aea7a2d: Status 404 returned error can't find the container with id 836754fc36dbe8d820084c1243a9c937f5e4f882a0f2d7149ca255c46aea7a2d
Sep 09 03:17:30.744048 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:30.743960    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:30.808003 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:30.807972    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:30.808373 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:30.808022    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:30.808373 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:30.808114    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:17:30.808373 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:30.808121    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:17:30.808373 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:30.808164    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-insights-runtime-extractor-tls podName:264d64b1-2bf7-4fea-acdd-0381a36e2e0c nodeName:}" failed. No retries permitted until 2026-09-09 03:17:32.808150924 +0000 UTC m=+269.944898767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-85tbf" (UID: "264d64b1-2bf7-4fea-acdd-0381a36e2e0c") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:17:30.808373 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:30.808188    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-rbac-proxy-cm podName:264d64b1-2bf7-4fea-acdd-0381a36e2e0c nodeName:}" failed. No retries permitted until 2026-09-09 03:17:32.808169825 +0000 UTC m=+269.944917672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-85tbf" (UID: "264d64b1-2bf7-4fea-acdd-0381a36e2e0c") : configmap "kube-rbac-proxy" not found
Sep 09 03:17:31.232179 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:31.232140    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-4hq44" event={"ID":"928cd3f5-4077-4ab0-bd5f-e191d30bb2ae","Type":"ContainerStarted","Data":"836754fc36dbe8d820084c1243a9c937f5e4f882a0f2d7149ca255c46aea7a2d"}
Sep 09 03:17:31.260357 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:31.260328    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sbmb7_def2fdc5-08ed-43a3-b9c1-b8b55e6444c5/dns-node-resolver/0.log"
Sep 09 03:17:31.412832 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:31.412787    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-snfbl\" (UID: \"aec25042-0f8f-4bda-a8bf-f1adb30990a6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"
Sep 09 03:17:31.413017 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:31.412975    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 03:17:31.413080 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:31.413050    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls podName:aec25042-0f8f-4bda-a8bf-f1adb30990a6 nodeName:}" failed. No retries permitted until 2026-09-09 03:17:39.413029545 +0000 UTC m=+276.549777406 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-snfbl" (UID: "aec25042-0f8f-4bda-a8bf-f1adb30990a6") : secret "cluster-monitoring-operator-tls" not found
Sep 09 03:17:32.060270 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:32.060239    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v47sj_0f4c4cf1-bf9c-499d-b25e-fe93588e6c58/node-ca/0.log"
Sep 09 03:17:32.826790 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:32.826747    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:32.826968 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:32.826848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:32.826968 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:32.826925    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:17:32.826968 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:32.826930    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:17:32.827059 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:32.826977    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-rbac-proxy-cm podName:264d64b1-2bf7-4fea-acdd-0381a36e2e0c nodeName:}" failed. No retries permitted until 2026-09-09 03:17:36.826963154 +0000 UTC m=+273.963710996 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-85tbf" (UID: "264d64b1-2bf7-4fea-acdd-0381a36e2e0c") : configmap "kube-rbac-proxy" not found
Sep 09 03:17:32.827059 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:32.826989    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-insights-runtime-extractor-tls podName:264d64b1-2bf7-4fea-acdd-0381a36e2e0c nodeName:}" failed. No retries permitted until 2026-09-09 03:17:36.826983584 +0000 UTC m=+273.963731426 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-85tbf" (UID: "264d64b1-2bf7-4fea-acdd-0381a36e2e0c") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:17:33.060660 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:33.060630    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h6lb8_90fe6e96-2fae-4cc7-97fb-1d46727849ed/serve-healthcheck-canary/0.log"
Sep 09 03:17:33.240286 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:33.240253    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-4hq44" event={"ID":"928cd3f5-4077-4ab0-bd5f-e191d30bb2ae","Type":"ContainerStarted","Data":"1e520cb8ee889fe095db6cb5ed340d97c5562ef262cd86fb7347eafe904d97a8"}
Sep 09 03:17:33.255968 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:33.255923    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-4hq44" podStartSLOduration=2.034909042 podStartE2EDuration="4.255908841s" totalImagesPullingTime="2.220999799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:17:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:17:33.254938341 +0000 UTC m=+270.391686205" watchObservedRunningTime="2026-09-09 03:17:33.255908841 +0000 UTC m=+270.392656704"
Sep 09 03:17:33.390144 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:33.390112    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:33.963628 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:33.963576    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-vhcbz"
Sep 09 03:17:33.963796 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:33.963666    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-vhcbz"
Sep 09 03:17:33.964112 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:33.964097    2560 scope.go:117] "RemoveContainer" containerID="14ce0a4405a187f5c4c43d5751486a816111af416519d300343dcb2b9872d0fa"
Sep 09 03:17:33.964330 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:33.964306    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-vhcbz_openshift-console-operator(5ebf6c83-e589-4833-994e-25ebfbb0b0a1)\"" pod="openshift-console-operator/console-operator-85b454874b-vhcbz" podUID="5ebf6c83-e589-4833-994e-25ebfbb0b0a1"
Sep 09 03:17:34.095873 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:34.095843    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:34.357407 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:34.357372    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:34.469587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:34.469557    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:34.676621 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:34.676523    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:34.777943 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:34.777913    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:34.878197 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:34.878166    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:35.764989 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:35.764936    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:36.165788 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:36.165758    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:36.366319 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:36.366265    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:36.572199 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:36.572114    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:36.768644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:36.768615    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:36.860302 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:36.860274    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:36.860453 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:36.860327    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:36.860453 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:36.860406    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:17:36.860577 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:36.860480    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-rbac-proxy-cm podName:264d64b1-2bf7-4fea-acdd-0381a36e2e0c nodeName:}" failed. No retries permitted until 2026-09-09 03:17:44.860459124 +0000 UTC m=+281.997206967 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-85tbf" (UID: "264d64b1-2bf7-4fea-acdd-0381a36e2e0c") : configmap "kube-rbac-proxy" not found
Sep 09 03:17:36.860577 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:36.860418    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:17:36.860577 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:36.860532    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-insights-runtime-extractor-tls podName:264d64b1-2bf7-4fea-acdd-0381a36e2e0c nodeName:}" failed. No retries permitted until 2026-09-09 03:17:44.860519837 +0000 UTC m=+281.997267680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-85tbf" (UID: "264d64b1-2bf7-4fea-acdd-0381a36e2e0c") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:17:36.976512 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:36.976480    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:37.448738 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:37.448707    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:37.969360 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:37.969323    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:38.069720 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:38.069671    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:38.169775 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:38.169746    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:38.366009 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:38.365979    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:38.605581 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:38.605547    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:38.770928 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:38.770829    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:39.478869 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:39.478825    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-snfbl\" (UID: \"aec25042-0f8f-4bda-a8bf-f1adb30990a6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"
Sep 09 03:17:39.479279 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:39.478964    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 03:17:39.479279 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:39.479037    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls podName:aec25042-0f8f-4bda-a8bf-f1adb30990a6 nodeName:}" failed. No retries permitted until 2026-09-09 03:17:55.479016641 +0000 UTC m=+292.615764503 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-snfbl" (UID: "aec25042-0f8f-4bda-a8bf-f1adb30990a6") : secret "cluster-monitoring-operator-tls" not found
Sep 09 03:17:39.769520 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:39.769425    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:40.173403 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:40.173374    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:40.363942 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:40.363900    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:40.639310 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:40.639274    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:40.690720 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:40.690688    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:40.969019 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:40.968939    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:41.144725 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:41.144694    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:41.567816 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:41.567772    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:41.966149 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:41.966111    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:42.169262 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:42.169187    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:42.775388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:42.775355    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:43.171426 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:43.171399    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:43.366440 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:43.366389    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:43.895693 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:43.895661    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:43.937322 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:43.937296    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn" podUID="6b8b2e70-6c05-4bfb-98ce-e18701db3e63"
Sep 09 03:17:43.937403 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:43.937296    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-c465c" podUID="a65c3985-522f-46f4-aeb2-7f12ae347816"
Sep 09 03:17:43.937403 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:43.937304    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d" podUID="87fed466-b71f-4bc5-b0e0-447ebecbaacb"
Sep 09 03:17:44.275296 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:44.275220    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:17:44.275296 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:44.275239    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:17:44.275296 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:44.275234    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c465c"
Sep 09 03:17:44.368059 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:44.368031    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:44.571036 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:44.570938    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:44.765629 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:44.765578    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:44.926245 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:44.926219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:44.926405 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:44.926371    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:44.926854 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:44.926838    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:44.928443 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:44.928424    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/264d64b1-2bf7-4fea-acdd-0381a36e2e0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85tbf\" (UID: \"264d64b1-2bf7-4fea-acdd-0381a36e2e0c\") " pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:45.005268 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:45.005242    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-85tbf"
Sep 09 03:17:45.273813 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:45.273722    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-85tbf"]
Sep 09 03:17:45.277273 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:17:45.277247    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264d64b1_2bf7_4fea_acdd_0381a36e2e0c.slice/crio-35dd7b94cec11ceadf479158e595fc8179e18e22aceab61979600e03c6913e6d WatchSource:0}: Error finding container 35dd7b94cec11ceadf479158e595fc8179e18e22aceab61979600e03c6913e6d: Status 404 returned error can't find the container with id 35dd7b94cec11ceadf479158e595fc8179e18e22aceab61979600e03c6913e6d
Sep 09 03:17:46.282958 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:46.282922    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-85tbf" event={"ID":"264d64b1-2bf7-4fea-acdd-0381a36e2e0c","Type":"ContainerStarted","Data":"57fc86267f3b47c96bd208f0b475a0c11878c2feabfa426719deb11ad9f5ee27"}
Sep 09 03:17:46.283390 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:46.282977    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-85tbf" event={"ID":"264d64b1-2bf7-4fea-acdd-0381a36e2e0c","Type":"ContainerStarted","Data":"7a981f7c1d7f30f74311c9f8b7b0586160c0c52c5642f2714c22ea52379be8bd"}
Sep 09 03:17:46.283390 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:46.282995    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-85tbf" event={"ID":"264d64b1-2bf7-4fea-acdd-0381a36e2e0c","Type":"ContainerStarted","Data":"35dd7b94cec11ceadf479158e595fc8179e18e22aceab61979600e03c6913e6d"}
Sep 09 03:17:47.246269 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:47.246230    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:17:47.246463 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:47.246292    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5jzgn\" (UID: \"6b8b2e70-6c05-4bfb-98ce-e18701db3e63\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:17:47.249022 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:47.248988    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b8b2e70-6c05-4bfb-98ce-e18701db3e63-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5jzgn\" (UID: \"6b8b2e70-6c05-4bfb-98ce-e18701db3e63\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:17:47.249153 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:47.249065    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls\") pod \"image-registry-94bb5b9d4-65f4d\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") " pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:17:47.279914 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:47.279888    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dmprq\""
Sep 09 03:17:47.279914 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:47.279888    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-gd747\""
Sep 09 03:17:47.286407 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:47.286375    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:17:47.286786 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:47.286452    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn"
Sep 09 03:17:47.347644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:47.347610    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:17:47.349854 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:47.349833    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a65c3985-522f-46f4-aeb2-7f12ae347816-metrics-tls\") pod \"dns-default-c465c\" (UID: \"a65c3985-522f-46f4-aeb2-7f12ae347816\") " pod="openshift-dns/dns-default-c465c"
Sep 09 03:17:47.357310 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:47.357291    2560 scope.go:117] "RemoveContainer" containerID="14ce0a4405a187f5c4c43d5751486a816111af416519d300343dcb2b9872d0fa"
Sep 09 03:17:47.579626 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:47.579542    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pxbsq\""
Sep 09 03:17:47.586917 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:47.586891    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c465c"
Sep 09 03:17:47.911575 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:47.911412    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-5jzgn"]
Sep 09 03:17:47.912780 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:47.912758    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c465c"]
Sep 09 03:17:47.913641 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:17:47.913617    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8b2e70_6c05_4bfb_98ce_e18701db3e63.slice/crio-c3950fd6b8e858601cbd2cd80971f9b2d1228ead897e3ae43c704a6113fbceb2 WatchSource:0}: Error finding container c3950fd6b8e858601cbd2cd80971f9b2d1228ead897e3ae43c704a6113fbceb2: Status 404 returned error can't find the container with id c3950fd6b8e858601cbd2cd80971f9b2d1228ead897e3ae43c704a6113fbceb2
Sep 09 03:17:47.915315 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:17:47.915291    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65c3985_522f_46f4_aeb2_7f12ae347816.slice/crio-43b59d329a5b561886e3cab52e4b943650ae31406579b8c3430f4aa96d61aaa4 WatchSource:0}: Error finding container 43b59d329a5b561886e3cab52e4b943650ae31406579b8c3430f4aa96d61aaa4: Status 404 returned error can't find the container with id 43b59d329a5b561886e3cab52e4b943650ae31406579b8c3430f4aa96d61aaa4
Sep 09 03:17:47.919983 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:47.919931    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-94bb5b9d4-65f4d"]
Sep 09 03:17:47.922209 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:17:47.922190    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87fed466_b71f_4bc5_b0e0_447ebecbaacb.slice/crio-13e4e3daf4e2e4a50d65cc48f547a5303424d8db6d8c10d5e08d7593dd53ddf8 WatchSource:0}: Error finding container 13e4e3daf4e2e4a50d65cc48f547a5303424d8db6d8c10d5e08d7593dd53ddf8: Status 404 returned error can't find the container with id 13e4e3daf4e2e4a50d65cc48f547a5303424d8db6d8c10d5e08d7593dd53ddf8
Sep 09 03:17:48.284689 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:48.284611    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:48.291382 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:48.291361    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:17:48.291821 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:48.291443    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-vhcbz" event={"ID":"5ebf6c83-e589-4833-994e-25ebfbb0b0a1","Type":"ContainerStarted","Data":"fea80c80807dfdb841dc5d60f5b2c6a20e38fce761e480ba3b07ece2f425a7b3"}
Sep 09 03:17:48.291821 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:48.291751    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-vhcbz"
Sep 09 03:17:48.292839 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:48.292808    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn" event={"ID":"6b8b2e70-6c05-4bfb-98ce-e18701db3e63","Type":"ContainerStarted","Data":"c3950fd6b8e858601cbd2cd80971f9b2d1228ead897e3ae43c704a6113fbceb2"}
Sep 09 03:17:48.295084 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:48.295060    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-85tbf" event={"ID":"264d64b1-2bf7-4fea-acdd-0381a36e2e0c","Type":"ContainerStarted","Data":"2a73d1b12d93dc63ef4a1a09e461a380aa73a8ca6d775a26403701ed9df1ec62"}
Sep 09 03:17:48.296665 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:48.296646    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-vhcbz"
Sep 09 03:17:48.297120 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:48.297100    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d" event={"ID":"87fed466-b71f-4bc5-b0e0-447ebecbaacb","Type":"ContainerStarted","Data":"bcaa58492f421d0fcdb2be0462a858e439288b4cd261f64da30ff1adea19a682"}
Sep 09 03:17:48.297224 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:48.297130    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d" event={"ID":"87fed466-b71f-4bc5-b0e0-447ebecbaacb","Type":"ContainerStarted","Data":"13e4e3daf4e2e4a50d65cc48f547a5303424d8db6d8c10d5e08d7593dd53ddf8"}
Sep 09 03:17:48.297224 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:48.297153    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:17:48.298288 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:48.298268    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c465c" event={"ID":"a65c3985-522f-46f4-aeb2-7f12ae347816","Type":"ContainerStarted","Data":"43b59d329a5b561886e3cab52e4b943650ae31406579b8c3430f4aa96d61aaa4"}
Sep 09 03:17:48.308883 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:48.308827    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-vhcbz" podStartSLOduration=22.326113992 podStartE2EDuration="25.308810544s" totalImagesPullingTime="2.982696552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:17:48.308482583 +0000 UTC m=+285.445230449" watchObservedRunningTime="2026-09-09 03:17:48.308810544 +0000 UTC m=+285.445558410"
Sep 09 03:17:48.351370 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:48.351340    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:48.366264 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:48.366217    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d" podStartSLOduration=284.366202187 podStartE2EDuration="4m44.366202187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:17:48.347095822 +0000 UTC m=+285.483843686" watchObservedRunningTime="2026-09-09 03:17:48.366202187 +0000 UTC m=+285.502950031"
Sep 09 03:17:48.366923 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:48.366894    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-85tbf" podStartSLOduration=17.069456628 podStartE2EDuration="19.366884522s" totalImagesPullingTime="2.297427894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:17:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:17:48.366339041 +0000 UTC m=+285.503086899" watchObservedRunningTime="2026-09-09 03:17:48.366884522 +0000 UTC m=+285.503632386"
Sep 09 03:17:48.414650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:48.414626    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:48.588846 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:48.588731    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:48.952102 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:48.952068    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:49.012457 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:49.012425    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:49.169916 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:49.169883    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:49.223545 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:49.223473    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:49.367014 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:49.366983    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:49.467698 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:49.467670    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:49.775138 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:49.775111    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:49.972359 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:49.972308    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:50.078412 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:50.078293    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:50.307309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:50.307271    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn" event={"ID":"6b8b2e70-6c05-4bfb-98ce-e18701db3e63","Type":"ContainerStarted","Data":"7d0fed9f1116c5f0fd3066e80a0478adcf5c4c65b8aaa8b774ffba4c06a1bbb4"}
Sep 09 03:17:50.323458 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:50.323395    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-5jzgn" podStartSLOduration=264.871155926 podStartE2EDuration="4m26.323379837s" totalImagesPullingTime="1.452223911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:13:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:17:50.323151356 +0000 UTC m=+287.459899221" watchObservedRunningTime="2026-09-09 03:17:50.323379837 +0000 UTC m=+287.460127701"
Sep 09 03:17:50.792923 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:50.792891    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:51.312396 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:51.312305    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c465c" event={"ID":"a65c3985-522f-46f4-aeb2-7f12ae347816","Type":"ContainerStarted","Data":"37a7042e03ac648fe344161bd241da83284178c6f3d53270e4473684086fc708"}
Sep 09 03:17:51.312396 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:51.312361    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c465c" event={"ID":"a65c3985-522f-46f4-aeb2-7f12ae347816","Type":"ContainerStarted","Data":"a325681c84f46a3b6439691bdae593d07508e81296f1ecb1b603a35808c4f27e"}
Sep 09 03:17:51.332474 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:51.332428    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c465c" podStartSLOduration=251.416890547 podStartE2EDuration="4m14.332413686s" totalImagesPullingTime="2.915523139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:17:51.330870873 +0000 UTC m=+288.467618737" watchObservedRunningTime="2026-09-09 03:17:51.332413686 +0000 UTC m=+288.469161551"
Sep 09 03:17:51.975159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:51.975124    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-k6ndc"]
Sep 09 03:17:51.997206 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:51.997166    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-k6ndc"]
Sep 09 03:17:51.997355 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:51.997281    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-k6ndc"
Sep 09 03:17:51.999984 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:51.999959    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 03:17:52.000113 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:51.999995    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-qncbh\""
Sep 09 03:17:52.000212 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:52.000198    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 03:17:52.029223 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:52.029184    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:52.089941 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:52.089913    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p64v4\" (UniqueName: \"kubernetes.io/projected/108cfd1b-5044-402e-8828-67b05550857a-kube-api-access-p64v4\") pod \"downloads-85d98c998f-k6ndc\" (UID: \"108cfd1b-5044-402e-8828-67b05550857a\") " pod="openshift-console/downloads-85d98c998f-k6ndc"
Sep 09 03:17:52.128741 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:52.128705    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:52.190509 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:52.190474    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p64v4\" (UniqueName: \"kubernetes.io/projected/108cfd1b-5044-402e-8828-67b05550857a-kube-api-access-p64v4\") pod \"downloads-85d98c998f-k6ndc\" (UID: \"108cfd1b-5044-402e-8828-67b05550857a\") " pod="openshift-console/downloads-85d98c998f-k6ndc"
Sep 09 03:17:52.216086 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:52.216051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p64v4\" (UniqueName: \"kubernetes.io/projected/108cfd1b-5044-402e-8828-67b05550857a-kube-api-access-p64v4\") pod \"downloads-85d98c998f-k6ndc\" (UID: \"108cfd1b-5044-402e-8828-67b05550857a\") " pod="openshift-console/downloads-85d98c998f-k6ndc"
Sep 09 03:17:52.273874 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:52.273769    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:52.306400 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:52.306353    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-k6ndc"
Sep 09 03:17:52.315384 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:52.315352    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c465c"
Sep 09 03:17:52.595237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:52.595146    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-k6ndc"]
Sep 09 03:17:52.597431 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:17:52.597400    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108cfd1b_5044_402e_8828_67b05550857a.slice/crio-c2cabc78fc48307600ae921e81ba1d20cbc7afa4084eca137f4f27567f78be42 WatchSource:0}: Error finding container c2cabc78fc48307600ae921e81ba1d20cbc7afa4084eca137f4f27567f78be42: Status 404 returned error can't find the container with id c2cabc78fc48307600ae921e81ba1d20cbc7afa4084eca137f4f27567f78be42
Sep 09 03:17:53.319370 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:53.319339    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-k6ndc" event={"ID":"108cfd1b-5044-402e-8828-67b05550857a","Type":"ContainerStarted","Data":"c2cabc78fc48307600ae921e81ba1d20cbc7afa4084eca137f4f27567f78be42"}
Sep 09 03:17:55.354307 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:55.354271    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hlmbb" podUID="f8508862-2ffb-4bd8-bb4f-67cd69a78075"
Sep 09 03:17:55.515190 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:55.515150    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-snfbl\" (UID: \"aec25042-0f8f-4bda-a8bf-f1adb30990a6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"
Sep 09 03:17:55.517831 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:55.517803    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aec25042-0f8f-4bda-a8bf-f1adb30990a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-snfbl\" (UID: \"aec25042-0f8f-4bda-a8bf-f1adb30990a6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"
Sep 09 03:17:55.774900 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:55.774818    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"
Sep 09 03:17:56.050570 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.050451    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl"]
Sep 09 03:17:56.053660 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:17:56.053630    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec25042_0f8f_4bda_a8bf_f1adb30990a6.slice/crio-cfb8a72ba212caabab1bb846b4bc3e10ee089a18391a7fce72aabf96965d575a WatchSource:0}: Error finding container cfb8a72ba212caabab1bb846b4bc3e10ee089a18391a7fce72aabf96965d575a: Status 404 returned error can't find the container with id cfb8a72ba212caabab1bb846b4bc3e10ee089a18391a7fce72aabf96965d575a
Sep 09 03:17:56.332006 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.331924    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl" event={"ID":"aec25042-0f8f-4bda-a8bf-f1adb30990a6","Type":"ContainerStarted","Data":"cfb8a72ba212caabab1bb846b4bc3e10ee089a18391a7fce72aabf96965d575a"}
Sep 09 03:17:56.657971 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.657930    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-758bf75bb8-22wb7"]
Sep 09 03:17:56.664618 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.664089    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.667835 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.667808    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 03:17:56.667977 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.667819    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 03:17:56.667977 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.667903    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 03:17:56.668280 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.668160    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 03:17:56.668667 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.668414    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rb7g8\""
Sep 09 03:17:56.668667 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.668611    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 03:17:56.670675 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.670624    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-758bf75bb8-22wb7"]
Sep 09 03:17:56.808216 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.808180    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:56.826358 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.826323    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvp98\" (UniqueName: \"kubernetes.io/projected/51a904dd-aa8a-4689-9635-e31d15b97dd0-kube-api-access-cvp98\") pod \"console-758bf75bb8-22wb7\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") " pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.826546 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.826402    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51a904dd-aa8a-4689-9635-e31d15b97dd0-console-oauth-config\") pod \"console-758bf75bb8-22wb7\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") " pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.826546 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.826428    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51a904dd-aa8a-4689-9635-e31d15b97dd0-console-config\") pod \"console-758bf75bb8-22wb7\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") " pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.826546 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.826464    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51a904dd-aa8a-4689-9635-e31d15b97dd0-service-ca\") pod \"console-758bf75bb8-22wb7\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") " pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.826546 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.826494    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51a904dd-aa8a-4689-9635-e31d15b97dd0-console-serving-cert\") pod \"console-758bf75bb8-22wb7\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") " pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.826546 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.826519    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51a904dd-aa8a-4689-9635-e31d15b97dd0-oauth-serving-cert\") pod \"console-758bf75bb8-22wb7\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") " pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.906793 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.906757    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:56.927467 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.927379    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvp98\" (UniqueName: \"kubernetes.io/projected/51a904dd-aa8a-4689-9635-e31d15b97dd0-kube-api-access-cvp98\") pod \"console-758bf75bb8-22wb7\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") " pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.927632 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.927548    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51a904dd-aa8a-4689-9635-e31d15b97dd0-console-oauth-config\") pod \"console-758bf75bb8-22wb7\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") " pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.927632 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.927582    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51a904dd-aa8a-4689-9635-e31d15b97dd0-console-config\") pod \"console-758bf75bb8-22wb7\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") " pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.927753 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.927640    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51a904dd-aa8a-4689-9635-e31d15b97dd0-service-ca\") pod \"console-758bf75bb8-22wb7\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") " pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.927753 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.927675    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51a904dd-aa8a-4689-9635-e31d15b97dd0-console-serving-cert\") pod \"console-758bf75bb8-22wb7\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") " pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.927753 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.927716    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51a904dd-aa8a-4689-9635-e31d15b97dd0-oauth-serving-cert\") pod \"console-758bf75bb8-22wb7\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") " pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.928469 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.928444    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51a904dd-aa8a-4689-9635-e31d15b97dd0-service-ca\") pod \"console-758bf75bb8-22wb7\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") " pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.928581 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.928477    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51a904dd-aa8a-4689-9635-e31d15b97dd0-oauth-serving-cert\") pod \"console-758bf75bb8-22wb7\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") " pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.928581 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.928489    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51a904dd-aa8a-4689-9635-e31d15b97dd0-console-config\") pod \"console-758bf75bb8-22wb7\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") " pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.930499 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.930479    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51a904dd-aa8a-4689-9635-e31d15b97dd0-console-oauth-config\") pod \"console-758bf75bb8-22wb7\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") " pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.930696 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.930678    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51a904dd-aa8a-4689-9635-e31d15b97dd0-console-serving-cert\") pod \"console-758bf75bb8-22wb7\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") " pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.936340 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.936317    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvp98\" (UniqueName: \"kubernetes.io/projected/51a904dd-aa8a-4689-9635-e31d15b97dd0-kube-api-access-cvp98\") pod \"console-758bf75bb8-22wb7\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") " pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:56.977381 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:56.977351    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:17:57.262147 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:57.262071    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-758bf75bb8-22wb7"]
Sep 09 03:17:57.730378 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:17:57.730340    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a904dd_aa8a_4689_9635_e31d15b97dd0.slice/crio-64bc40880d4dfe28464d87c37a3bd6f180a7220054150cc16ea2d32ac5ed91c9 WatchSource:0}: Error finding container 64bc40880d4dfe28464d87c37a3bd6f180a7220054150cc16ea2d32ac5ed91c9: Status 404 returned error can't find the container with id 64bc40880d4dfe28464d87c37a3bd6f180a7220054150cc16ea2d32ac5ed91c9
Sep 09 03:17:58.104426 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:58.104391    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:58.204871 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:58.204793    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:58.259997 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:58.259963    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hbmnj"]
Sep 09 03:17:58.263317 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:58.263293    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hbmnj"
Sep 09 03:17:58.265965 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:58.265776    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 09 03:17:58.265965 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:58.265876    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-z4qr8\""
Sep 09 03:17:58.273067 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:58.273004    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hbmnj"]
Sep 09 03:17:58.277902 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:58.277874    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:58.344764 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:58.344726    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758bf75bb8-22wb7" event={"ID":"51a904dd-aa8a-4689-9635-e31d15b97dd0","Type":"ContainerStarted","Data":"64bc40880d4dfe28464d87c37a3bd6f180a7220054150cc16ea2d32ac5ed91c9"}
Sep 09 03:17:58.346692 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:58.346660    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl" event={"ID":"aec25042-0f8f-4bda-a8bf-f1adb30990a6","Type":"ContainerStarted","Data":"1c4f4c8d46a7f44f2167a968b487dc63c92eb6ec8ea4ce5f58c40e7b8c28e09b"}
Sep 09 03:17:58.367874 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:58.367571    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-snfbl" podStartSLOduration=33.646073358 podStartE2EDuration="35.367549908s" totalImagesPullingTime="1.72147655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:17:58.366832144 +0000 UTC m=+295.503580010" watchObservedRunningTime="2026-09-09 03:17:58.367549908 +0000 UTC m=+295.504297775"
Sep 09 03:17:58.445338 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:58.445298    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3536d6ff-720b-4bf3-b4c6-c4f75a2cffeb-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-hbmnj\" (UID: \"3536d6ff-720b-4bf3-b4c6-c4f75a2cffeb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hbmnj"
Sep 09 03:17:58.547297 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:58.546815    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3536d6ff-720b-4bf3-b4c6-c4f75a2cffeb-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-hbmnj\" (UID: \"3536d6ff-720b-4bf3-b4c6-c4f75a2cffeb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hbmnj"
Sep 09 03:17:58.547297 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:58.546996    2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 09 03:17:58.547297 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:17:58.547066    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3536d6ff-720b-4bf3-b4c6-c4f75a2cffeb-tls-certificates podName:3536d6ff-720b-4bf3-b4c6-c4f75a2cffeb nodeName:}" failed. No retries permitted until 2026-09-09 03:17:59.047043842 +0000 UTC m=+296.183791690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/3536d6ff-720b-4bf3-b4c6-c4f75a2cffeb-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-hbmnj" (UID: "3536d6ff-720b-4bf3-b4c6-c4f75a2cffeb") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 09 03:17:58.611397 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:58.611360    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:59.051247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:59.051209    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3536d6ff-720b-4bf3-b4c6-c4f75a2cffeb-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-hbmnj\" (UID: \"3536d6ff-720b-4bf3-b4c6-c4f75a2cffeb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hbmnj"
Sep 09 03:17:59.054270 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:59.054237    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3536d6ff-720b-4bf3-b4c6-c4f75a2cffeb-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-hbmnj\" (UID: \"3536d6ff-720b-4bf3-b4c6-c4f75a2cffeb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hbmnj"
Sep 09 03:17:59.175157 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:59.175087    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hbmnj"
Sep 09 03:17:59.192440 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:59.192399    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:59.247414 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:59.247366    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:17:59.460192 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:17:59.459849    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hbmnj"]
Sep 09 03:17:59.463356 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:17:59.463328    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3536d6ff_720b_4bf3_b4c6_c4f75a2cffeb.slice/crio-3a2f8e0feb53f72680b1a15679a40798ef37ca419cb2b0db22b3bb7b0d2f7c6c WatchSource:0}: Error finding container 3a2f8e0feb53f72680b1a15679a40798ef37ca419cb2b0db22b3bb7b0d2f7c6c: Status 404 returned error can't find the container with id 3a2f8e0feb53f72680b1a15679a40798ef37ca419cb2b0db22b3bb7b0d2f7c6c
Sep 09 03:18:00.358178 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:00.358100    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hbmnj" event={"ID":"3536d6ff-720b-4bf3-b4c6-c4f75a2cffeb","Type":"ContainerStarted","Data":"3a2f8e0feb53f72680b1a15679a40798ef37ca419cb2b0db22b3bb7b0d2f7c6c"}
Sep 09 03:18:01.365104 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:01.365065    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758bf75bb8-22wb7" event={"ID":"51a904dd-aa8a-4689-9635-e31d15b97dd0","Type":"ContainerStarted","Data":"d174ec03461f2ebecafed22cdbdeb81597beb0dc55ae37a1c9f5d8c6368aa836"}
Sep 09 03:18:01.385426 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:01.385343    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-758bf75bb8-22wb7" podStartSLOduration=2.1509826690000002 podStartE2EDuration="5.385319809s" totalImagesPullingTime="3.23433714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:18:01.384581393 +0000 UTC m=+298.521329253" watchObservedRunningTime="2026-09-09 03:18:01.385319809 +0000 UTC m=+298.522067674"
Sep 09 03:18:02.322064 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:02.322030    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c465c"
Sep 09 03:18:02.371977 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:02.371942    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hbmnj" event={"ID":"3536d6ff-720b-4bf3-b4c6-c4f75a2cffeb","Type":"ContainerStarted","Data":"e7bddba485defb2a8b3ec4def56f3edb5979a8202fc1072cfe09c9873ec8229b"}
Sep 09 03:18:02.372481 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:02.372110    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hbmnj"
Sep 09 03:18:02.377288 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:02.377255    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hbmnj"
Sep 09 03:18:02.392300 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:02.392250    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hbmnj" podStartSLOduration=2.203150152 podStartE2EDuration="4.392231835s" totalImagesPullingTime="2.189081683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:18:02.390856439 +0000 UTC m=+299.527604305" watchObservedRunningTime="2026-09-09 03:18:02.392231835 +0000 UTC m=+299.528979701"
Sep 09 03:18:03.295181 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.295145    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:03.324439 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.324408    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-hcs7l"]
Sep 09 03:18:03.328869 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.328847    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-hcs7l"
Sep 09 03:18:03.331299 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.331277    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 03:18:03.332632 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.332460    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 03:18:03.332632 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.332478    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 03:18:03.332632 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.332460    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9zkr6\""
Sep 09 03:18:03.336571 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.336545    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-hcs7l"]
Sep 09 03:18:03.398691 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.398647    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:03.492883 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.492850    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a94c8e5a-97f8-4a15-bf3d-b73758df0235-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-hcs7l\" (UID: \"a94c8e5a-97f8-4a15-bf3d-b73758df0235\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hcs7l"
Sep 09 03:18:03.493065 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.492889    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pp8k\" (UniqueName: \"kubernetes.io/projected/a94c8e5a-97f8-4a15-bf3d-b73758df0235-kube-api-access-9pp8k\") pod \"prometheus-operator-6f4946db56-hcs7l\" (UID: \"a94c8e5a-97f8-4a15-bf3d-b73758df0235\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hcs7l"
Sep 09 03:18:03.493065 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.492948    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a94c8e5a-97f8-4a15-bf3d-b73758df0235-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-hcs7l\" (UID: \"a94c8e5a-97f8-4a15-bf3d-b73758df0235\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hcs7l"
Sep 09 03:18:03.493065 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.492974    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a94c8e5a-97f8-4a15-bf3d-b73758df0235-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-hcs7l\" (UID: \"a94c8e5a-97f8-4a15-bf3d-b73758df0235\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hcs7l"
Sep 09 03:18:03.523503 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.523467    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:03.593681 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.593571    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a94c8e5a-97f8-4a15-bf3d-b73758df0235-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-hcs7l\" (UID: \"a94c8e5a-97f8-4a15-bf3d-b73758df0235\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hcs7l"
Sep 09 03:18:03.593681 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.593648    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a94c8e5a-97f8-4a15-bf3d-b73758df0235-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-hcs7l\" (UID: \"a94c8e5a-97f8-4a15-bf3d-b73758df0235\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hcs7l"
Sep 09 03:18:03.593911 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.593740    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a94c8e5a-97f8-4a15-bf3d-b73758df0235-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-hcs7l\" (UID: \"a94c8e5a-97f8-4a15-bf3d-b73758df0235\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hcs7l"
Sep 09 03:18:03.593911 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.593768    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pp8k\" (UniqueName: \"kubernetes.io/projected/a94c8e5a-97f8-4a15-bf3d-b73758df0235-kube-api-access-9pp8k\") pod \"prometheus-operator-6f4946db56-hcs7l\" (UID: \"a94c8e5a-97f8-4a15-bf3d-b73758df0235\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hcs7l"
Sep 09 03:18:03.594363 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.594340    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a94c8e5a-97f8-4a15-bf3d-b73758df0235-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-hcs7l\" (UID: \"a94c8e5a-97f8-4a15-bf3d-b73758df0235\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hcs7l"
Sep 09 03:18:03.596552 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.596528    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a94c8e5a-97f8-4a15-bf3d-b73758df0235-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-hcs7l\" (UID: \"a94c8e5a-97f8-4a15-bf3d-b73758df0235\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hcs7l"
Sep 09 03:18:03.596661 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.596556    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a94c8e5a-97f8-4a15-bf3d-b73758df0235-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-hcs7l\" (UID: \"a94c8e5a-97f8-4a15-bf3d-b73758df0235\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hcs7l"
Sep 09 03:18:03.602762 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.602715    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pp8k\" (UniqueName: \"kubernetes.io/projected/a94c8e5a-97f8-4a15-bf3d-b73758df0235-kube-api-access-9pp8k\") pod \"prometheus-operator-6f4946db56-hcs7l\" (UID: \"a94c8e5a-97f8-4a15-bf3d-b73758df0235\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-hcs7l"
Sep 09 03:18:03.641441 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:03.641405    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-hcs7l"
Sep 09 03:18:05.731175 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:05.731137    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:06.409672 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.409575    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f5d7f97f6-p8x5h"]
Sep 09 03:18:06.413144 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.413112    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.422579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.422162    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 03:18:06.423967 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.423940    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f5d7f97f6-p8x5h"]
Sep 09 03:18:06.512692 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.512643    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-console-config\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.512870 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.512708    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ww7g\" (UniqueName: \"kubernetes.io/projected/b1308509-b56e-4f2d-a21a-2687d7f649e1-kube-api-access-7ww7g\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.512870 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.512748    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-oauth-serving-cert\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.512870 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.512853    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-service-ca\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.513042 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.512893    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1308509-b56e-4f2d-a21a-2687d7f649e1-console-serving-cert\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.513042 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.512927    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1308509-b56e-4f2d-a21a-2687d7f649e1-console-oauth-config\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.513042 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.512986    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-trusted-ca-bundle\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.613778 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.613738    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-trusted-ca-bundle\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.613963 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.613884    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-console-config\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.613963 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.613941    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ww7g\" (UniqueName: \"kubernetes.io/projected/b1308509-b56e-4f2d-a21a-2687d7f649e1-kube-api-access-7ww7g\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.614074 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.613983    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-oauth-serving-cert\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.614074 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.614043    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-service-ca\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.614173 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.614074    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1308509-b56e-4f2d-a21a-2687d7f649e1-console-serving-cert\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.614173 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.614107    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1308509-b56e-4f2d-a21a-2687d7f649e1-console-oauth-config\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.614682 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.614634    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-console-config\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.614813 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.614789    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-trusted-ca-bundle\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.615006 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.614984    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-service-ca\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.615082 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.615046    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-oauth-serving-cert\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.616950 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.616929    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1308509-b56e-4f2d-a21a-2687d7f649e1-console-serving-cert\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.617207 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.617184    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1308509-b56e-4f2d-a21a-2687d7f649e1-console-oauth-config\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.622997 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.622973    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ww7g\" (UniqueName: \"kubernetes.io/projected/b1308509-b56e-4f2d-a21a-2687d7f649e1-kube-api-access-7ww7g\") pod \"console-6f5d7f97f6-p8x5h\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") " pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.725913 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.725823    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:06.977748 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.977665    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:18:06.977748 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.977730    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:18:06.982956 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:06.982930    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:18:07.291987 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:07.291901    2560 patch_prober.go:28] interesting pod/image-registry-94bb5b9d4-65f4d container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 03:18:07.291987 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:07.291963    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d" podUID="87fed466-b71f-4bc5-b0e0-447ebecbaacb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:18:07.397297 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:07.397265    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:18:07.419475 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:07.419443    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:08.353281 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:08.353245    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:18:09.305681 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:09.305655    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:18:09.341397 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:09.341371    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:18:09.355170 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:09.355113    2560 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 03:18:09.369913 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:09.368036    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:18:09.598427 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:09.598339    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:09.610463 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:09.610427    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-hcs7l"]
Sep 09 03:18:09.610560 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:09.610495    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f5d7f97f6-p8x5h"]
Sep 09 03:18:09.610647 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:18:09.610561    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1308509_b56e_4f2d_a21a_2687d7f649e1.slice/crio-78484aeba7d5b05f27d69fad2affabe1d68134f011cb84537ceeeed2e3c10b81 WatchSource:0}: Error finding container 78484aeba7d5b05f27d69fad2affabe1d68134f011cb84537ceeeed2e3c10b81: Status 404 returned error can't find the container with id 78484aeba7d5b05f27d69fad2affabe1d68134f011cb84537ceeeed2e3c10b81
Sep 09 03:18:09.611373 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:09.611352    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:18:09.693775 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:09.693711    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:10.408081 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:10.408043    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5d7f97f6-p8x5h" event={"ID":"b1308509-b56e-4f2d-a21a-2687d7f649e1","Type":"ContainerStarted","Data":"27bf5b239f39cb539cbcbcfbf0ed3a97625f16c1f75b5451d9b85b452c5c6745"}
Sep 09 03:18:10.408560 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:10.408104    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5d7f97f6-p8x5h" event={"ID":"b1308509-b56e-4f2d-a21a-2687d7f649e1","Type":"ContainerStarted","Data":"78484aeba7d5b05f27d69fad2affabe1d68134f011cb84537ceeeed2e3c10b81"}
Sep 09 03:18:10.409932 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:10.409896    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-hcs7l" event={"ID":"a94c8e5a-97f8-4a15-bf3d-b73758df0235","Type":"ContainerStarted","Data":"088df162196e538b68d69a86bbaa0c8dae0a69b9d998039f4184dff9acb0a72d"}
Sep 09 03:18:10.411790 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:10.411761    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-k6ndc" event={"ID":"108cfd1b-5044-402e-8828-67b05550857a","Type":"ContainerStarted","Data":"370317ac44aa522a9a98784a0be54572c335161d46fff259d905ae17fdd67237"}
Sep 09 03:18:10.412011 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:10.411980    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-k6ndc"
Sep 09 03:18:10.421183 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:10.421160    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-k6ndc"
Sep 09 03:18:10.429456 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:10.429407    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f5d7f97f6-p8x5h" podStartSLOduration=4.429393194 podStartE2EDuration="4.429393194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:18:10.429102165 +0000 UTC m=+307.565850031" watchObservedRunningTime="2026-09-09 03:18:10.429393194 +0000 UTC m=+307.566141059"
Sep 09 03:18:10.447540 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:10.447494    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-k6ndc" podStartSLOduration=2.630158877 podStartE2EDuration="19.447479704s" totalImagesPullingTime="16.817320827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:18:10.446420079 +0000 UTC m=+307.583167945" watchObservedRunningTime="2026-09-09 03:18:10.447479704 +0000 UTC m=+307.584227569"
Sep 09 03:18:10.464184 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:10.464148    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:11.016073 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:11.016026    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:11.117689 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:11.117627    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:11.485023 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:11.484989    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:11.600335 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:11.600294    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:11.697565 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:11.697535    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:12.292186 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:12.292096    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:12.422943 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:12.422886    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-hcs7l" event={"ID":"a94c8e5a-97f8-4a15-bf3d-b73758df0235","Type":"ContainerStarted","Data":"3dd835c18ab4bb941cc969781db31e5621ea1c28e9f55c44e014170b03061a34"}
Sep 09 03:18:12.423130 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:12.422977    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-hcs7l" event={"ID":"a94c8e5a-97f8-4a15-bf3d-b73758df0235","Type":"ContainerStarted","Data":"9e84f5af233a4f226c69a704c9f4e042f9986be66c79ba77d516a2643b5a5b44"}
Sep 09 03:18:12.443927 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:12.443867    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-hcs7l" podStartSLOduration=7.142297422 podStartE2EDuration="9.443847674s" totalImagesPullingTime="2.301550252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:18:12.442622876 +0000 UTC m=+309.579370744" watchObservedRunningTime="2026-09-09 03:18:12.443847674 +0000 UTC m=+309.580595547"
Sep 09 03:18:14.397802 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.396082    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-94bb5b9d4-65f4d"]
Sep 09 03:18:14.436077 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.436008    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:14.538199 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.538154    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:14.642936 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.642900    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:14.721617 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.721523    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"]
Sep 09 03:18:14.775939 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.775900    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:14.785405 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.785378    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"]
Sep 09 03:18:14.785552 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.785430    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wq9cn"]
Sep 09 03:18:14.785632 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.785566    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:14.788378 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.788324    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-f7695\""
Sep 09 03:18:14.789791 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.789620    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 09 03:18:14.789791 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.789750    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 09 03:18:14.789973 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.789930    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 09 03:18:14.805084 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.805063    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.808693 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.808665    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 03:18:14.808797 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.808663    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 03:18:14.808797 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.808752    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-z8kcn\""
Sep 09 03:18:14.808915 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.808900    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 03:18:14.869554 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.869491    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:14.888057 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.888027    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b8c76543-c8a2-4f13-891a-127f0afdb670-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-qs94b\" (UID: \"b8c76543-c8a2-4f13-891a-127f0afdb670\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:14.888198 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.888077    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-sys\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.888198 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.888108    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8c76543-c8a2-4f13-891a-127f0afdb670-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-qs94b\" (UID: \"b8c76543-c8a2-4f13-891a-127f0afdb670\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:14.888198 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.888164    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b8c76543-c8a2-4f13-891a-127f0afdb670-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-qs94b\" (UID: \"b8c76543-c8a2-4f13-891a-127f0afdb670\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:14.888198 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.888191    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8c76543-c8a2-4f13-891a-127f0afdb670-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-qs94b\" (UID: \"b8c76543-c8a2-4f13-891a-127f0afdb670\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:14.888409 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.888234    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.888409 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.888291    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-metrics-client-ca\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.888409 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.888323    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-root\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.888409 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.888359    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-node-exporter-tls\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.888409 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.888380    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-node-exporter-wtmp\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.888409 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.888408    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-node-exporter-textfile\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.888746 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.888440    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdm2d\" (UniqueName: \"kubernetes.io/projected/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-kube-api-access-hdm2d\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.888746 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.888468    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-node-exporter-accelerators-collector-config\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.888746 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.888498    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8c76543-c8a2-4f13-891a-127f0afdb670-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-qs94b\" (UID: \"b8c76543-c8a2-4f13-891a-127f0afdb670\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:14.888746 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.888543    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nhqh\" (UniqueName: \"kubernetes.io/projected/b8c76543-c8a2-4f13-891a-127f0afdb670-kube-api-access-7nhqh\") pod \"kube-state-metrics-74cb57df7-qs94b\" (UID: \"b8c76543-c8a2-4f13-891a-127f0afdb670\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:14.971935 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.971856    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:14.990010 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.989936    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-node-exporter-accelerators-collector-config\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.990010 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.989981    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8c76543-c8a2-4f13-891a-127f0afdb670-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-qs94b\" (UID: \"b8c76543-c8a2-4f13-891a-127f0afdb670\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:14.990010 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.990009    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nhqh\" (UniqueName: \"kubernetes.io/projected/b8c76543-c8a2-4f13-891a-127f0afdb670-kube-api-access-7nhqh\") pod \"kube-state-metrics-74cb57df7-qs94b\" (UID: \"b8c76543-c8a2-4f13-891a-127f0afdb670\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:14.990272 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.990040    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b8c76543-c8a2-4f13-891a-127f0afdb670-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-qs94b\" (UID: \"b8c76543-c8a2-4f13-891a-127f0afdb670\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:14.990272 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.990077    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-sys\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.990272 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.990105    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8c76543-c8a2-4f13-891a-127f0afdb670-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-qs94b\" (UID: \"b8c76543-c8a2-4f13-891a-127f0afdb670\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:14.990272 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.990159    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b8c76543-c8a2-4f13-891a-127f0afdb670-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-qs94b\" (UID: \"b8c76543-c8a2-4f13-891a-127f0afdb670\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:14.990272 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.990191    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8c76543-c8a2-4f13-891a-127f0afdb670-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-qs94b\" (UID: \"b8c76543-c8a2-4f13-891a-127f0afdb670\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:14.990272 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.990236    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.990568 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.990281    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-metrics-client-ca\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.990568 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.990324    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-root\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.990568 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.990348    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-node-exporter-tls\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.990568 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.990371    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-node-exporter-wtmp\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.990568 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.990407    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-node-exporter-textfile\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.990568 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.990444    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdm2d\" (UniqueName: \"kubernetes.io/projected/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-kube-api-access-hdm2d\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.990568 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.990539    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-node-exporter-accelerators-collector-config\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.991183 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.991156    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8c76543-c8a2-4f13-891a-127f0afdb670-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-qs94b\" (UID: \"b8c76543-c8a2-4f13-891a-127f0afdb670\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:14.991301 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.991227    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-sys\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.991652 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.991633    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b8c76543-c8a2-4f13-891a-127f0afdb670-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-qs94b\" (UID: \"b8c76543-c8a2-4f13-891a-127f0afdb670\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:14.991791 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.991713    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-node-exporter-wtmp\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.991986 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.991962    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-node-exporter-textfile\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.992077 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.992057    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b8c76543-c8a2-4f13-891a-127f0afdb670-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-qs94b\" (UID: \"b8c76543-c8a2-4f13-891a-127f0afdb670\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:14.992138 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.992115    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-root\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.992500 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.992451    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-metrics-client-ca\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.997624 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.995011    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:14.997624 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.995427    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8c76543-c8a2-4f13-891a-127f0afdb670-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-qs94b\" (UID: \"b8c76543-c8a2-4f13-891a-127f0afdb670\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:14.997624 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.997070    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8c76543-c8a2-4f13-891a-127f0afdb670-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-qs94b\" (UID: \"b8c76543-c8a2-4f13-891a-127f0afdb670\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:14.997624 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:14.997430    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-node-exporter-tls\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:15.004348 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.004250    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdm2d\" (UniqueName: \"kubernetes.io/projected/bb1b3371-60f4-4ef6-ae93-f55d583b2eb6-kube-api-access-hdm2d\") pod \"node-exporter-wq9cn\" (UID: \"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6\") " pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:15.007722 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.006884    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nhqh\" (UniqueName: \"kubernetes.io/projected/b8c76543-c8a2-4f13-891a-127f0afdb670-kube-api-access-7nhqh\") pod \"kube-state-metrics-74cb57df7-qs94b\" (UID: \"b8c76543-c8a2-4f13-891a-127f0afdb670\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:15.074373 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.074331    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:15.099314 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.099285    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"
Sep 09 03:18:15.119741 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.119680    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wq9cn"
Sep 09 03:18:15.173207 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.173125    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:15.273212 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.273093    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:15.371940 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.371911    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:15.422270 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.422147    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-qs94b"]
Sep 09 03:18:15.424792 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:18:15.424763    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c76543_c8a2_4f13_891a_127f0afdb670.slice/crio-87b4793104acde707efd1af3fe9810c8a201577b46bae08804d2b3eb9bf99b84 WatchSource:0}: Error finding container 87b4793104acde707efd1af3fe9810c8a201577b46bae08804d2b3eb9bf99b84: Status 404 returned error can't find the container with id 87b4793104acde707efd1af3fe9810c8a201577b46bae08804d2b3eb9bf99b84
Sep 09 03:18:15.447969 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.447911    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wq9cn" event={"ID":"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6","Type":"ContainerStarted","Data":"08bba0e97561f69b6f79bcf04c6adfce935866a1a49c488a311b7dcac426c1db"}
Sep 09 03:18:15.452227 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.452193    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b" event={"ID":"b8c76543-c8a2-4f13-891a-127f0afdb670","Type":"ContainerStarted","Data":"87b4793104acde707efd1af3fe9810c8a201577b46bae08804d2b3eb9bf99b84"}
Sep 09 03:18:15.822217 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.821098    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:15.915692 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.915660    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:15.920532 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.920503    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 03:18:15.943377 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.941899    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:15.945620 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.944654    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 03:18:15.945620 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.945164    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 03:18:15.945979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.945959    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 03:18:15.949358 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.947161    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 03:18:15.949358 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.947345    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 03:18:15.949358 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.947501    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 03:18:15.949358 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.947695    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 03:18:15.949358 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.947874    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jhpr6\""
Sep 09 03:18:15.949358 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.948057    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 03:18:15.949358 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.948243    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 03:18:15.949358 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:15.948473    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 03:18:16.008306 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.007703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.008306 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.007774    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fa800a9-04b3-49d9-8748-89b7afb841b8-config-out\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.008306 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.007804    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-config-volume\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.008306 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.007832    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fa800a9-04b3-49d9-8748-89b7afb841b8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.008306 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.007857    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.008306 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.007896    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.008306 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.007931    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-web-config\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.008306 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.007954    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2kq9\" (UniqueName: \"kubernetes.io/projected/0fa800a9-04b3-49d9-8748-89b7afb841b8-kube-api-access-k2kq9\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.008306 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.008006    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fa800a9-04b3-49d9-8748-89b7afb841b8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.008306 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.008032    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.008306 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.008081    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0fa800a9-04b3-49d9-8748-89b7afb841b8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.008306 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.008107    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa800a9-04b3-49d9-8748-89b7afb841b8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.008306 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.008136    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.108886 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.108644    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fa800a9-04b3-49d9-8748-89b7afb841b8-config-out\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.108886 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.108693    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-config-volume\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.108886 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.108723    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fa800a9-04b3-49d9-8748-89b7afb841b8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.108886 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.108745    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.108886 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.108780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.108886 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.108814    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-web-config\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.108886 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.108836    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2kq9\" (UniqueName: \"kubernetes.io/projected/0fa800a9-04b3-49d9-8748-89b7afb841b8-kube-api-access-k2kq9\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.108886 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.108870    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fa800a9-04b3-49d9-8748-89b7afb841b8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.108886 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.108888    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.109466 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.108936    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0fa800a9-04b3-49d9-8748-89b7afb841b8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.109466 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.108965    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa800a9-04b3-49d9-8748-89b7afb841b8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.109466 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.108993    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.109466 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.109027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.113878 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.110627    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fa800a9-04b3-49d9-8748-89b7afb841b8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.113878 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.110972    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0fa800a9-04b3-49d9-8748-89b7afb841b8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.113878 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:18:16.111065    2560 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 09 03:18:16.113878 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:18:16.111130    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-secret-alertmanager-main-tls podName:0fa800a9-04b3-49d9-8748-89b7afb841b8 nodeName:}" failed. No retries permitted until 2026-09-09 03:18:16.611111703 +0000 UTC m=+313.747859561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "0fa800a9-04b3-49d9-8748-89b7afb841b8") : secret "alertmanager-main-tls" not found
Sep 09 03:18:16.113878 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.112528    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fa800a9-04b3-49d9-8748-89b7afb841b8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.114568 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.114513    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.115278 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.115238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fa800a9-04b3-49d9-8748-89b7afb841b8-config-out\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.119476 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.117562    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-config-volume\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.119793 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.119749    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.120005 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.119984    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.120005 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.119995    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.120415 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.120370    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fa800a9-04b3-49d9-8748-89b7afb841b8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.120512 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.120459    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-web-config\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.123616 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.123555    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2kq9\" (UniqueName: \"kubernetes.io/projected/0fa800a9-04b3-49d9-8748-89b7afb841b8-kube-api-access-k2kq9\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.233847 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.233810    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:16.614192 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.614148    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.616986 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.616956    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0fa800a9-04b3-49d9-8748-89b7afb841b8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0fa800a9-04b3-49d9-8748-89b7afb841b8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:16.726726 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.726550    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:16.726726 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.726636    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:16.733242 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.733091    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:16.861876 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:16.861837    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 03:18:17.469388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:17.469358    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:18:17.520801 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:17.520764    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-758bf75bb8-22wb7"]
Sep 09 03:18:17.696845 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:17.696814    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:17.793016 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:17.792939    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:17.876477 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:17.876444    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 03:18:17.881085 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:18:17.881041    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa800a9_04b3_49d9_8748_89b7afb841b8.slice/crio-7d4c69a8adf145dd7a56e0fffcaa38fab57965b6854730bdff2ebe627c3fb1dd WatchSource:0}: Error finding container 7d4c69a8adf145dd7a56e0fffcaa38fab57965b6854730bdff2ebe627c3fb1dd: Status 404 returned error can't find the container with id 7d4c69a8adf145dd7a56e0fffcaa38fab57965b6854730bdff2ebe627c3fb1dd
Sep 09 03:18:17.952894 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:17.952732    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:18.225156 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:18.225126    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:18.473164 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:18.473124    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b" event={"ID":"b8c76543-c8a2-4f13-891a-127f0afdb670","Type":"ContainerStarted","Data":"b8b74174a6e4fded56dd3223e5bd19f2b15d6c048711e98ee6fce24137fcae6a"}
Sep 09 03:18:18.473331 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:18.473199    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b" event={"ID":"b8c76543-c8a2-4f13-891a-127f0afdb670","Type":"ContainerStarted","Data":"be456a7738a5b820b76fd3bd544fc4e9ce3b21f48210bd7ac5e37767994400df"}
Sep 09 03:18:18.473331 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:18.473237    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b" event={"ID":"b8c76543-c8a2-4f13-891a-127f0afdb670","Type":"ContainerStarted","Data":"75f05c7eb73bb9f4386c71ba07c31ccbf83ceaa579f6d42f512f8644ba97e9f7"}
Sep 09 03:18:18.475041 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:18.475009    2560 generic.go:356] "Generic (PLEG): container finished" podID="bb1b3371-60f4-4ef6-ae93-f55d583b2eb6" containerID="0e5b9f857c494c1e946260fce1a271d592bb7c3c6ea68f47ab6b964571660ef0" exitCode=0
Sep 09 03:18:18.475184 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:18.475093    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wq9cn" event={"ID":"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6","Type":"ContainerDied","Data":"0e5b9f857c494c1e946260fce1a271d592bb7c3c6ea68f47ab6b964571660ef0"}
Sep 09 03:18:18.476685 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:18.476624    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fa800a9-04b3-49d9-8748-89b7afb841b8","Type":"ContainerStarted","Data":"7d4c69a8adf145dd7a56e0fffcaa38fab57965b6854730bdff2ebe627c3fb1dd"}
Sep 09 03:18:18.494013 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:18.493968    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-qs94b" podStartSLOduration=2.349128325 podStartE2EDuration="4.493954827s" totalImagesPullingTime="2.144826502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:18:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:18:18.492528941 +0000 UTC m=+315.629276822" watchObservedRunningTime="2026-09-09 03:18:18.493954827 +0000 UTC m=+315.630702692"
Sep 09 03:18:18.957526 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:18.957500    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:19.054644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:19.054610    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:19.485019 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:19.484930    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:19.486809 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:19.486773    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wq9cn" event={"ID":"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6","Type":"ContainerStarted","Data":"9f791fda66bd86dd86f1982baf9685e6b337a07969143ddf2119674fa737241b"}
Sep 09 03:18:19.486939 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:19.486833    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wq9cn" event={"ID":"bb1b3371-60f4-4ef6-ae93-f55d583b2eb6","Type":"ContainerStarted","Data":"25a68057a2c80d6b645c51fcffebba602356e3e84a69654f21922c382ec763c5"}
Sep 09 03:18:19.488476 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:19.488452    2560 generic.go:356] "Generic (PLEG): container finished" podID="0fa800a9-04b3-49d9-8748-89b7afb841b8" containerID="820d6f8a0d242ce6567c192a72bfcc8081c26af21b349de0a3731c83704597cf" exitCode=0
Sep 09 03:18:19.488607 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:19.488506    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fa800a9-04b3-49d9-8748-89b7afb841b8","Type":"ContainerDied","Data":"820d6f8a0d242ce6567c192a72bfcc8081c26af21b349de0a3731c83704597cf"}
Sep 09 03:18:19.513688 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:19.513634    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wq9cn" podStartSLOduration=2.987369555 podStartE2EDuration="5.513619269s" totalImagesPullingTime="2.395925824s" totalInitContainerRuntime="130.32389ms" isStatefulPod=true podCreationTimestamp="2026-09-09 03:18:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:18:19.512258454 +0000 UTC m=+316.649006320" watchObservedRunningTime="2026-09-09 03:18:19.513619269 +0000 UTC m=+316.650367145"
Sep 09 03:18:19.583272 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:19.583239    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:19.682864 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:19.682837    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:20.025686 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.025651    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:20.087350 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.087307    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:20.484969 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.484935    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59fdb9cc59-9dkqj"]
Sep 09 03:18:20.501742 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.501713    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:20.505936 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.505913    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59fdb9cc59-9dkqj"]
Sep 09 03:18:20.506072 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.506001    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.549536 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.549510    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqclr\" (UniqueName: \"kubernetes.io/projected/861c585a-cfcc-4851-b878-b25241d19a46-kube-api-access-xqclr\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.549684 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.549589    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-oauth-serving-cert\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.549684 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.549646    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/861c585a-cfcc-4851-b878-b25241d19a46-console-oauth-config\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.549791 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.549695    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/861c585a-cfcc-4851-b878-b25241d19a46-console-serving-cert\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.549791 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.549741    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-trusted-ca-bundle\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.549920 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.549897    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-service-ca\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.550137 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.550117    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-console-config\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.567808 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.567769    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:20.651015 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.650981    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-oauth-serving-cert\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.651179 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.651033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/861c585a-cfcc-4851-b878-b25241d19a46-console-oauth-config\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.651179 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.651065    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/861c585a-cfcc-4851-b878-b25241d19a46-console-serving-cert\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.651179 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.651101    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-trusted-ca-bundle\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.651179 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.651166    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-service-ca\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.651382 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.651205    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-console-config\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.651382 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.651240    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqclr\" (UniqueName: \"kubernetes.io/projected/861c585a-cfcc-4851-b878-b25241d19a46-kube-api-access-xqclr\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.651809 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.651786    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-oauth-serving-cert\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.651976 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.651947    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-service-ca\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.652043 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.652022    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-trusted-ca-bundle\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.652138 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.652106    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-console-config\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.653685 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.653666    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/861c585a-cfcc-4851-b878-b25241d19a46-console-oauth-config\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.653794 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.653700    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/861c585a-cfcc-4851-b878-b25241d19a46-console-serving-cert\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.660548 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.660525    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqclr\" (UniqueName: \"kubernetes.io/projected/861c585a-cfcc-4851-b878-b25241d19a46-kube-api-access-xqclr\") pod \"console-59fdb9cc59-9dkqj\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") " pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:20.815312 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:20.815218    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:21.008297 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.008178    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:21.011037 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.010640    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 03:18:21.036691 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.036657    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 03:18:21.037005 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.036835    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.039885 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.039653    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 03:18:21.039885 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.039705    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 03:18:21.039885 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.039653    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 03:18:21.040913 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.040285    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 03:18:21.040913 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.040313    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 03:18:21.040913 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.040483    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 03:18:21.040913 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.040550    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4m8sp\""
Sep 09 03:18:21.040913 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.040778    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 03:18:21.041217 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.041130    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 03:18:21.043968 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.041480    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 03:18:21.043968 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.041729    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 03:18:21.043968 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.041745    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 03:18:21.044184 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.044111    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 03:18:21.045379 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.044860    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e1au442l0pd5e\""
Sep 09 03:18:21.045379 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.044868    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 03:18:21.045379 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.045291    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 03:18:21.053609 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.047764    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 03:18:21.156191 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156159    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.156191 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156194    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.156401 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156211    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.156401 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156235    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/334da5a2-1482-4597-8a4b-18b11c9cf3dd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.156401 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156284    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-config\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.156401 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156320    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.156401 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156335    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.156401 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156352    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.156401 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.156401 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156395    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-web-config\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.156740 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156459    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/334da5a2-1482-4597-8a4b-18b11c9cf3dd-config-out\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.156740 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156487    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.156740 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.156740 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156541    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.156740 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156611    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.156740 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156636    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.156740 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156651    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.156740 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156716    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/334da5a2-1482-4597-8a4b-18b11c9cf3dd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.157031 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156756    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.157031 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.156784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h8jt\" (UniqueName: \"kubernetes.io/projected/334da5a2-1482-4597-8a4b-18b11c9cf3dd-kube-api-access-2h8jt\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.234696 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.234653    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59fdb9cc59-9dkqj"]
Sep 09 03:18:21.237212 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.237062    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:21.257336 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.257308    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.257457 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.257341    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.257457 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.257363    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.257457 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.257387    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.257636 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.257534    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.257636 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.257577    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.257869 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.257831    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/334da5a2-1482-4597-8a4b-18b11c9cf3dd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.258031 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.258010    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.258152 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.258129    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.258309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.258134    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h8jt\" (UniqueName: \"kubernetes.io/projected/334da5a2-1482-4597-8a4b-18b11c9cf3dd-kube-api-access-2h8jt\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.258427 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.258363    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.258427 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.258401    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.258552 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.258427    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.258552 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.258451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/334da5a2-1482-4597-8a4b-18b11c9cf3dd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.258552 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.258200    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.258552 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.258188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.258552 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.258490    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-config\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.260010 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.258560    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.260010 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.258588    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.260010 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.258710    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.260010 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.258732    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.260010 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.258792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-web-config\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.260010 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.258825    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/334da5a2-1482-4597-8a4b-18b11c9cf3dd-config-out\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.260010 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.259225    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.260010 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.259559    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.262436 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.262412    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.263012 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.262824    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/334da5a2-1482-4597-8a4b-18b11c9cf3dd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.263109 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.263084    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.263314 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.263269    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/334da5a2-1482-4597-8a4b-18b11c9cf3dd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.263388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.263351    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.264656 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.263629    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-config\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.264656 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.263835    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.266097 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.265518    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.266097 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.265646    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.266097 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.265668    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.266097 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.266045    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.266340 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.266115    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/334da5a2-1482-4597-8a4b-18b11c9cf3dd-web-config\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.266923 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.266905    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/334da5a2-1482-4597-8a4b-18b11c9cf3dd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.267218 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.267201    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h8jt\" (UniqueName: \"kubernetes.io/projected/334da5a2-1482-4597-8a4b-18b11c9cf3dd-kube-api-access-2h8jt\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.267451 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.267432    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/334da5a2-1482-4597-8a4b-18b11c9cf3dd-config-out\") pod \"prometheus-k8s-0\" (UID: \"334da5a2-1482-4597-8a4b-18b11c9cf3dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.297193 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.297165    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:21.314176 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:18:21.314148    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861c585a_cfcc_4851_b878_b25241d19a46.slice/crio-27f0e9af491b7e915fcbe84671d9c79a414a57036464c0d21002a940fbdccfa6 WatchSource:0}: Error finding container 27f0e9af491b7e915fcbe84671d9c79a414a57036464c0d21002a940fbdccfa6: Status 404 returned error can't find the container with id 27f0e9af491b7e915fcbe84671d9c79a414a57036464c0d21002a940fbdccfa6
Sep 09 03:18:21.358609 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.358570    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:21.502419 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.502369    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fa800a9-04b3-49d9-8748-89b7afb841b8","Type":"ContainerStarted","Data":"311f0b247d9a5b48c94256bea41bad32abbe3884a27e69dd6ac123e075991bee"}
Sep 09 03:18:21.504142 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.504118    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59fdb9cc59-9dkqj" event={"ID":"861c585a-cfcc-4851-b878-b25241d19a46","Type":"ContainerStarted","Data":"1b3f3218f3fae66ab1d91cbc1351824d917168ca6edf4dd25b3feca9faa88562"}
Sep 09 03:18:21.504347 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.504165    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59fdb9cc59-9dkqj" event={"ID":"861c585a-cfcc-4851-b878-b25241d19a46","Type":"ContainerStarted","Data":"27f0e9af491b7e915fcbe84671d9c79a414a57036464c0d21002a940fbdccfa6"}
Sep 09 03:18:21.528499 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.528453    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59fdb9cc59-9dkqj" podStartSLOduration=1.528437348 podStartE2EDuration="1.528437348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:18:21.527480873 +0000 UTC m=+318.664228739" watchObservedRunningTime="2026-09-09 03:18:21.528437348 +0000 UTC m=+318.665185213"
Sep 09 03:18:21.705085 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:21.705063    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 03:18:21.707127 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:18:21.707100    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334da5a2_1482_4597_8a4b_18b11c9cf3dd.slice/crio-49ca74c87194190b62976c4fa76c6c7385bbafa2b0e3bf6ace97969b2a1c0034 WatchSource:0}: Error finding container 49ca74c87194190b62976c4fa76c6c7385bbafa2b0e3bf6ace97969b2a1c0034: Status 404 returned error can't find the container with id 49ca74c87194190b62976c4fa76c6c7385bbafa2b0e3bf6ace97969b2a1c0034
Sep 09 03:18:22.511332 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:22.511296    2560 generic.go:356] "Generic (PLEG): container finished" podID="334da5a2-1482-4597-8a4b-18b11c9cf3dd" containerID="87cac9ff987c444bfa572025f7b9a11238181b0ee35cc9e845a9cc06a214ac59" exitCode=0
Sep 09 03:18:22.511821 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:22.511385    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"334da5a2-1482-4597-8a4b-18b11c9cf3dd","Type":"ContainerDied","Data":"87cac9ff987c444bfa572025f7b9a11238181b0ee35cc9e845a9cc06a214ac59"}
Sep 09 03:18:22.511821 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:22.511460    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"334da5a2-1482-4597-8a4b-18b11c9cf3dd","Type":"ContainerStarted","Data":"49ca74c87194190b62976c4fa76c6c7385bbafa2b0e3bf6ace97969b2a1c0034"}
Sep 09 03:18:22.520556 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:22.520524    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fa800a9-04b3-49d9-8748-89b7afb841b8","Type":"ContainerStarted","Data":"2faf6490a66e22ec7101116543cf4118bf04c49484986d52f54979f94bfdf991"}
Sep 09 03:18:22.520718 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:22.520662    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fa800a9-04b3-49d9-8748-89b7afb841b8","Type":"ContainerStarted","Data":"658c2376e31fdd5efbd07b9d5bca9539e4f13cd23ed5d97a6d419fc5da5dd5e5"}
Sep 09 03:18:22.520718 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:22.520708    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fa800a9-04b3-49d9-8748-89b7afb841b8","Type":"ContainerStarted","Data":"16017e743a937a3578c3f2c8a5628d2c3189112495c2b446a97872a52239e965"}
Sep 09 03:18:22.520831 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:22.520754    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fa800a9-04b3-49d9-8748-89b7afb841b8","Type":"ContainerStarted","Data":"5b51b37cf70f898869dd9e261fb6f6c10b73d1c23f42ea476eda5b7fcc3c46bc"}
Sep 09 03:18:23.529556 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:23.529517    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0fa800a9-04b3-49d9-8748-89b7afb841b8","Type":"ContainerStarted","Data":"e76fd061723fec852ba556290418a17287a832ef0296cf691020cd7aee3301d0"}
Sep 09 03:18:23.564018 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:23.563960    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.268378757 podStartE2EDuration="8.563940421s" totalImagesPullingTime="4.272674751s" totalInitContainerRuntime="22.886913ms" isStatefulPod=false podCreationTimestamp="2026-09-09 03:18:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:18:23.561296984 +0000 UTC m=+320.698044849" watchObservedRunningTime="2026-09-09 03:18:23.563940421 +0000 UTC m=+320.700688286"
Sep 09 03:18:24.869361 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:24.869323    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59fdb9cc59-9dkqj"]
Sep 09 03:18:24.905670 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:24.905641    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-776df47b56-lpj2g"]
Sep 09 03:18:24.939723 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:24.939669    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-776df47b56-lpj2g"]
Sep 09 03:18:24.939903 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:24.939812    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:24.995257 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:24.995228    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvxv4\" (UniqueName: \"kubernetes.io/projected/117902b2-b67c-48e2-88c0-ac7a2560d122-kube-api-access-jvxv4\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:24.995431 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:24.995282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-service-ca\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:24.995431 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:24.995366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/117902b2-b67c-48e2-88c0-ac7a2560d122-console-serving-cert\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:24.995431 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:24.995388    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/117902b2-b67c-48e2-88c0-ac7a2560d122-console-oauth-config\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:24.995575 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:24.995430    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-console-config\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:24.995575 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:24.995486    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-trusted-ca-bundle\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:24.995575 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:24.995512    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-oauth-serving-cert\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:25.097739 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.096080    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvxv4\" (UniqueName: \"kubernetes.io/projected/117902b2-b67c-48e2-88c0-ac7a2560d122-kube-api-access-jvxv4\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:25.097739 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.096141    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-service-ca\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:25.097739 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.096244    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/117902b2-b67c-48e2-88c0-ac7a2560d122-console-serving-cert\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:25.097739 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.096269    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/117902b2-b67c-48e2-88c0-ac7a2560d122-console-oauth-config\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:25.097739 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.096292    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-console-config\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:25.097739 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.096321    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-trusted-ca-bundle\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:25.097739 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.096346    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-oauth-serving-cert\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:25.097739 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.097635    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-trusted-ca-bundle\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:25.098266 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.097792    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-service-ca\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:25.098266 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.097922    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-console-config\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:25.098394 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.098276    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-oauth-serving-cert\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:25.099824 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.099800    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/117902b2-b67c-48e2-88c0-ac7a2560d122-console-oauth-config\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:25.100687 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.100646    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/117902b2-b67c-48e2-88c0-ac7a2560d122-console-serving-cert\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:25.110328 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.109403    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvxv4\" (UniqueName: \"kubernetes.io/projected/117902b2-b67c-48e2-88c0-ac7a2560d122-kube-api-access-jvxv4\") pod \"console-776df47b56-lpj2g\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") " pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:25.252682 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.252510    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:25.342196 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.342155    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:25.406792 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.406754    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:25.839589 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.839510    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-776df47b56-lpj2g"]
Sep 09 03:18:25.864680 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.864647    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:25.928303 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:18:25.928265    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117902b2_b67c_48e2_88c0_ac7a2560d122.slice/crio-8e5bde5a8ff48e75beb2424a6a11006e614b92ef83918a13da338bc8548c4780 WatchSource:0}: Error finding container 8e5bde5a8ff48e75beb2424a6a11006e614b92ef83918a13da338bc8548c4780: Status 404 returned error can't find the container with id 8e5bde5a8ff48e75beb2424a6a11006e614b92ef83918a13da338bc8548c4780
Sep 09 03:18:25.931797 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:25.931774    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:26.545696 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:26.545659    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"334da5a2-1482-4597-8a4b-18b11c9cf3dd","Type":"ContainerStarted","Data":"be2fab9a81264053a5da92bfe9adf20bb2e0a84c8e324b61f9cc345abb86e05c"}
Sep 09 03:18:26.545883 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:26.545748    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"334da5a2-1482-4597-8a4b-18b11c9cf3dd","Type":"ContainerStarted","Data":"9432716e10a51b6a9c90573c5c73e815c8d2a24e58f2056e2058a48393d0efc9"}
Sep 09 03:18:26.547154 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:26.547132    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776df47b56-lpj2g" event={"ID":"117902b2-b67c-48e2-88c0-ac7a2560d122","Type":"ContainerStarted","Data":"ae94cc7081fedea87737927ec3d07c2e2feca1d79d039a8e3e08e6b56c84f1db"}
Sep 09 03:18:26.547241 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:26.547167    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776df47b56-lpj2g" event={"ID":"117902b2-b67c-48e2-88c0-ac7a2560d122","Type":"ContainerStarted","Data":"8e5bde5a8ff48e75beb2424a6a11006e614b92ef83918a13da338bc8548c4780"}
Sep 09 03:18:26.567873 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:26.567823    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-776df47b56-lpj2g" podStartSLOduration=2.567808202 podStartE2EDuration="2.567808202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:18:26.56681665 +0000 UTC m=+323.703564534" watchObservedRunningTime="2026-09-09 03:18:26.567808202 +0000 UTC m=+323.704556063"
Sep 09 03:18:27.229069 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:27.229038    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:27.326498 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:27.326460    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:29.563502 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:29.563466    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"334da5a2-1482-4597-8a4b-18b11c9cf3dd","Type":"ContainerStarted","Data":"1b2474cbe2ad8ef20bde8a1af969560259a25bad05cee17f1e8a898b6f37a8bb"}
Sep 09 03:18:29.563887 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:29.563536    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"334da5a2-1482-4597-8a4b-18b11c9cf3dd","Type":"ContainerStarted","Data":"e5be449f3c0f12ad890172404584adacc821bbedb9e0ef90a90e8e930d6c537d"}
Sep 09 03:18:29.563887 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:29.563565    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"334da5a2-1482-4597-8a4b-18b11c9cf3dd","Type":"ContainerStarted","Data":"57019152972d9e6f569b70646095a883af215d8b651ee8168b478e734135061a"}
Sep 09 03:18:29.563887 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:29.563611    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"334da5a2-1482-4597-8a4b-18b11c9cf3dd","Type":"ContainerStarted","Data":"3edf8a04224a329c2f4aef26710819728500af7c9f99579f4d91cb69b79ba8fe"}
Sep 09 03:18:29.590343 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:29.590296    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.843091179 podStartE2EDuration="9.590282226s" totalImagesPullingTime="5.744060016s" totalInitContainerRuntime="3.131031ms" isStatefulPod=false podCreationTimestamp="2026-09-09 03:18:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:18:29.589148322 +0000 UTC m=+326.725896187" watchObservedRunningTime="2026-09-09 03:18:29.590282226 +0000 UTC m=+326.727030091"
Sep 09 03:18:30.816123 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:30.816084    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:31.359718 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:31.359691    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:18:33.957606 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:33.957556    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:34.022206 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:34.022173    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:35.253241 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:35.253207    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:35.253655 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:35.253286    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:35.258190 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:35.258168    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:35.594524 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:35.594450    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:18:35.642151 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:35.642121    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f5d7f97f6-p8x5h"]
Sep 09 03:18:35.773870 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:35.773831    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:36.026871 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:36.026839    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:36.127209 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:36.127180    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:38.605675 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:38.605640    2560 generic.go:356] "Generic (PLEG): container finished" podID="6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593" containerID="c6976c45417e675b04729bfe31777a8e3d449955cf3eb90c2cd184e5206242bf" exitCode=0
Sep 09 03:18:38.606047 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:38.605713    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc" event={"ID":"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593","Type":"ContainerDied","Data":"c6976c45417e675b04729bfe31777a8e3d449955cf3eb90c2cd184e5206242bf"}
Sep 09 03:18:38.606107 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:38.606093    2560 scope.go:117] "RemoveContainer" containerID="c6976c45417e675b04729bfe31777a8e3d449955cf3eb90c2cd184e5206242bf"
Sep 09 03:18:39.428171 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.428101    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d" podUID="87fed466-b71f-4bc5-b0e0-447ebecbaacb" containerName="registry" containerID="cri-o://bcaa58492f421d0fcdb2be0462a858e439288b4cd261f64da30ff1adea19a682" gracePeriod=30
Sep 09 03:18:39.612845 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.612816    2560 generic.go:356] "Generic (PLEG): container finished" podID="87fed466-b71f-4bc5-b0e0-447ebecbaacb" containerID="bcaa58492f421d0fcdb2be0462a858e439288b4cd261f64da30ff1adea19a682" exitCode=0
Sep 09 03:18:39.613378 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.612889    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d" event={"ID":"87fed466-b71f-4bc5-b0e0-447ebecbaacb","Type":"ContainerDied","Data":"bcaa58492f421d0fcdb2be0462a858e439288b4cd261f64da30ff1adea19a682"}
Sep 09 03:18:39.615068 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.615043    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-jtgvc" event={"ID":"6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593","Type":"ContainerStarted","Data":"8e80b177c64845b5080329daba4cecddfc12d7fd5f4f648fd5432e326546cdfc"}
Sep 09 03:18:39.677739 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.677714    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:18:39.841977 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.841899    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls\") pod \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") "
Sep 09 03:18:39.841977 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.841931    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87fed466-b71f-4bc5-b0e0-447ebecbaacb-trusted-ca\") pod \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") "
Sep 09 03:18:39.841977 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.841975    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-bound-sa-token\") pod \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") "
Sep 09 03:18:39.842252 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.842006    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnfs6\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-kube-api-access-vnfs6\") pod \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") "
Sep 09 03:18:39.842252 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.842026    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87fed466-b71f-4bc5-b0e0-447ebecbaacb-image-registry-private-configuration\") pod \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") "
Sep 09 03:18:39.842252 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.842053    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87fed466-b71f-4bc5-b0e0-447ebecbaacb-ca-trust-extracted\") pod \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") "
Sep 09 03:18:39.842252 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.842085    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87fed466-b71f-4bc5-b0e0-447ebecbaacb-installation-pull-secrets\") pod \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") "
Sep 09 03:18:39.842450 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.842292    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-certificates\") pod \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\" (UID: \"87fed466-b71f-4bc5-b0e0-447ebecbaacb\") "
Sep 09 03:18:39.842450 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.842332    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87fed466-b71f-4bc5-b0e0-447ebecbaacb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "87fed466-b71f-4bc5-b0e0-447ebecbaacb" (UID: "87fed466-b71f-4bc5-b0e0-447ebecbaacb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:18:39.842809 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.842716    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87fed466-b71f-4bc5-b0e0-447ebecbaacb-trusted-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:39.843005 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.842979    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "87fed466-b71f-4bc5-b0e0-447ebecbaacb" (UID: "87fed466-b71f-4bc5-b0e0-447ebecbaacb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:18:39.844399 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.844365    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "87fed466-b71f-4bc5-b0e0-447ebecbaacb" (UID: "87fed466-b71f-4bc5-b0e0-447ebecbaacb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:18:39.844766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.844738    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87fed466-b71f-4bc5-b0e0-447ebecbaacb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "87fed466-b71f-4bc5-b0e0-447ebecbaacb" (UID: "87fed466-b71f-4bc5-b0e0-447ebecbaacb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:18:39.844766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.844747    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-kube-api-access-vnfs6" (OuterVolumeSpecName: "kube-api-access-vnfs6") pod "87fed466-b71f-4bc5-b0e0-447ebecbaacb" (UID: "87fed466-b71f-4bc5-b0e0-447ebecbaacb"). InnerVolumeSpecName "kube-api-access-vnfs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:18:39.844938 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.844811    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "87fed466-b71f-4bc5-b0e0-447ebecbaacb" (UID: "87fed466-b71f-4bc5-b0e0-447ebecbaacb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:18:39.845048 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.844938    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87fed466-b71f-4bc5-b0e0-447ebecbaacb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "87fed466-b71f-4bc5-b0e0-447ebecbaacb" (UID: "87fed466-b71f-4bc5-b0e0-447ebecbaacb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:18:39.851104 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.851076    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87fed466-b71f-4bc5-b0e0-447ebecbaacb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "87fed466-b71f-4bc5-b0e0-447ebecbaacb" (UID: "87fed466-b71f-4bc5-b0e0-447ebecbaacb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:18:39.943375 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.943335    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:39.943375 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.943369    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-bound-sa-token\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:39.943375 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.943378    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnfs6\" (UniqueName: \"kubernetes.io/projected/87fed466-b71f-4bc5-b0e0-447ebecbaacb-kube-api-access-vnfs6\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:39.943619 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.943388    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87fed466-b71f-4bc5-b0e0-447ebecbaacb-image-registry-private-configuration\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:39.943619 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.943398    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87fed466-b71f-4bc5-b0e0-447ebecbaacb-ca-trust-extracted\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:39.943619 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.943406    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87fed466-b71f-4bc5-b0e0-447ebecbaacb-installation-pull-secrets\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:39.943619 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:39.943415    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87fed466-b71f-4bc5-b0e0-447ebecbaacb-registry-certificates\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:40.035426 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:40.035389    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:40.621014 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:40.620978    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d" event={"ID":"87fed466-b71f-4bc5-b0e0-447ebecbaacb","Type":"ContainerDied","Data":"13e4e3daf4e2e4a50d65cc48f547a5303424d8db6d8c10d5e08d7593dd53ddf8"}
Sep 09 03:18:40.621365 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:40.621018    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-94bb5b9d4-65f4d"
Sep 09 03:18:40.621365 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:40.621046    2560 scope.go:117] "RemoveContainer" containerID="bcaa58492f421d0fcdb2be0462a858e439288b4cd261f64da30ff1adea19a682"
Sep 09 03:18:40.642114 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:40.642091    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-94bb5b9d4-65f4d"]
Sep 09 03:18:40.646359 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:40.646335    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-94bb5b9d4-65f4d"]
Sep 09 03:18:41.358301 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:41.358268    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87fed466-b71f-4bc5-b0e0-447ebecbaacb" path="/var/lib/kubelet/pods/87fed466-b71f-4bc5-b0e0-447ebecbaacb/volumes"
Sep 09 03:18:42.550405 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.550360    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-758bf75bb8-22wb7" podUID="51a904dd-aa8a-4689-9635-e31d15b97dd0" containerName="console" containerID="cri-o://d174ec03461f2ebecafed22cdbdeb81597beb0dc55ae37a1c9f5d8c6368aa836" gracePeriod=15
Sep 09 03:18:42.833791 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.833734    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:18:42.870866 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.870834    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51a904dd-aa8a-4689-9635-e31d15b97dd0-console-config\") pod \"51a904dd-aa8a-4689-9635-e31d15b97dd0\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") "
Sep 09 03:18:42.871035 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.870872    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51a904dd-aa8a-4689-9635-e31d15b97dd0-oauth-serving-cert\") pod \"51a904dd-aa8a-4689-9635-e31d15b97dd0\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") "
Sep 09 03:18:42.871035 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.870914    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51a904dd-aa8a-4689-9635-e31d15b97dd0-console-oauth-config\") pod \"51a904dd-aa8a-4689-9635-e31d15b97dd0\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") "
Sep 09 03:18:42.871035 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.870931    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51a904dd-aa8a-4689-9635-e31d15b97dd0-service-ca\") pod \"51a904dd-aa8a-4689-9635-e31d15b97dd0\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") "
Sep 09 03:18:42.871035 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.870951    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51a904dd-aa8a-4689-9635-e31d15b97dd0-console-serving-cert\") pod \"51a904dd-aa8a-4689-9635-e31d15b97dd0\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") "
Sep 09 03:18:42.871035 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.871005    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvp98\" (UniqueName: \"kubernetes.io/projected/51a904dd-aa8a-4689-9635-e31d15b97dd0-kube-api-access-cvp98\") pod \"51a904dd-aa8a-4689-9635-e31d15b97dd0\" (UID: \"51a904dd-aa8a-4689-9635-e31d15b97dd0\") "
Sep 09 03:18:42.871325 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.871287    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a904dd-aa8a-4689-9635-e31d15b97dd0-console-config" (OuterVolumeSpecName: "console-config") pod "51a904dd-aa8a-4689-9635-e31d15b97dd0" (UID: "51a904dd-aa8a-4689-9635-e31d15b97dd0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:18:42.871406 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.871380    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a904dd-aa8a-4689-9635-e31d15b97dd0-service-ca" (OuterVolumeSpecName: "service-ca") pod "51a904dd-aa8a-4689-9635-e31d15b97dd0" (UID: "51a904dd-aa8a-4689-9635-e31d15b97dd0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:18:42.871660 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.871635    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a904dd-aa8a-4689-9635-e31d15b97dd0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "51a904dd-aa8a-4689-9635-e31d15b97dd0" (UID: "51a904dd-aa8a-4689-9635-e31d15b97dd0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:18:42.873188 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.873166    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51a904dd-aa8a-4689-9635-e31d15b97dd0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "51a904dd-aa8a-4689-9635-e31d15b97dd0" (UID: "51a904dd-aa8a-4689-9635-e31d15b97dd0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:18:42.873284 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.873257    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51a904dd-aa8a-4689-9635-e31d15b97dd0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "51a904dd-aa8a-4689-9635-e31d15b97dd0" (UID: "51a904dd-aa8a-4689-9635-e31d15b97dd0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:18:42.873557 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.873542    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a904dd-aa8a-4689-9635-e31d15b97dd0-kube-api-access-cvp98" (OuterVolumeSpecName: "kube-api-access-cvp98") pod "51a904dd-aa8a-4689-9635-e31d15b97dd0" (UID: "51a904dd-aa8a-4689-9635-e31d15b97dd0"). InnerVolumeSpecName "kube-api-access-cvp98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:18:42.972101 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.972067    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51a904dd-aa8a-4689-9635-e31d15b97dd0-console-config\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:42.972101 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.972094    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51a904dd-aa8a-4689-9635-e31d15b97dd0-oauth-serving-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:42.972101 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.972105    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51a904dd-aa8a-4689-9635-e31d15b97dd0-console-oauth-config\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:42.972101 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.972113    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51a904dd-aa8a-4689-9635-e31d15b97dd0-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:42.972350 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.972121    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51a904dd-aa8a-4689-9635-e31d15b97dd0-console-serving-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:42.972350 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:42.972131    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvp98\" (UniqueName: \"kubernetes.io/projected/51a904dd-aa8a-4689-9635-e31d15b97dd0-kube-api-access-cvp98\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:43.636843 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:43.636753    2560 generic.go:356] "Generic (PLEG): container finished" podID="51a904dd-aa8a-4689-9635-e31d15b97dd0" containerID="d174ec03461f2ebecafed22cdbdeb81597beb0dc55ae37a1c9f5d8c6368aa836" exitCode=0
Sep 09 03:18:43.636843 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:43.636802    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758bf75bb8-22wb7"
Sep 09 03:18:43.637261 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:43.636841    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758bf75bb8-22wb7" event={"ID":"51a904dd-aa8a-4689-9635-e31d15b97dd0","Type":"ContainerDied","Data":"d174ec03461f2ebecafed22cdbdeb81597beb0dc55ae37a1c9f5d8c6368aa836"}
Sep 09 03:18:43.637261 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:43.636899    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758bf75bb8-22wb7" event={"ID":"51a904dd-aa8a-4689-9635-e31d15b97dd0","Type":"ContainerDied","Data":"64bc40880d4dfe28464d87c37a3bd6f180a7220054150cc16ea2d32ac5ed91c9"}
Sep 09 03:18:43.637261 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:43.636920    2560 scope.go:117] "RemoveContainer" containerID="d174ec03461f2ebecafed22cdbdeb81597beb0dc55ae37a1c9f5d8c6368aa836"
Sep 09 03:18:43.644830 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:43.644813    2560 scope.go:117] "RemoveContainer" containerID="d174ec03461f2ebecafed22cdbdeb81597beb0dc55ae37a1c9f5d8c6368aa836"
Sep 09 03:18:43.645104 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:18:43.645082    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d174ec03461f2ebecafed22cdbdeb81597beb0dc55ae37a1c9f5d8c6368aa836\": container with ID starting with d174ec03461f2ebecafed22cdbdeb81597beb0dc55ae37a1c9f5d8c6368aa836 not found: ID does not exist" containerID="d174ec03461f2ebecafed22cdbdeb81597beb0dc55ae37a1c9f5d8c6368aa836"
Sep 09 03:18:43.645167 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:43.645117    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d174ec03461f2ebecafed22cdbdeb81597beb0dc55ae37a1c9f5d8c6368aa836"} err="failed to get container status \"d174ec03461f2ebecafed22cdbdeb81597beb0dc55ae37a1c9f5d8c6368aa836\": rpc error: code = NotFound desc = could not find container \"d174ec03461f2ebecafed22cdbdeb81597beb0dc55ae37a1c9f5d8c6368aa836\": container with ID starting with d174ec03461f2ebecafed22cdbdeb81597beb0dc55ae37a1c9f5d8c6368aa836 not found: ID does not exist"
Sep 09 03:18:43.652994 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:43.652970    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-758bf75bb8-22wb7"]
Sep 09 03:18:43.658653 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:43.658635    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-758bf75bb8-22wb7"]
Sep 09 03:18:45.358317 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:45.358282    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51a904dd-aa8a-4689-9635-e31d15b97dd0" path="/var/lib/kubelet/pods/51a904dd-aa8a-4689-9635-e31d15b97dd0/volumes"
Sep 09 03:18:48.956972 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:48.956937    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:49.021056 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:49.021026    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:49.891735 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:49.891690    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59fdb9cc59-9dkqj" podUID="861c585a-cfcc-4851-b878-b25241d19a46" containerName="console" containerID="cri-o://1b3f3218f3fae66ab1d91cbc1351824d917168ca6edf4dd25b3feca9faa88562" gracePeriod=15
Sep 09 03:18:50.182866 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.182835    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:50.239323 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.239291    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-service-ca\") pod \"861c585a-cfcc-4851-b878-b25241d19a46\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") "
Sep 09 03:18:50.239504 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.239369    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/861c585a-cfcc-4851-b878-b25241d19a46-console-oauth-config\") pod \"861c585a-cfcc-4851-b878-b25241d19a46\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") "
Sep 09 03:18:50.239504 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.239407    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/861c585a-cfcc-4851-b878-b25241d19a46-console-serving-cert\") pod \"861c585a-cfcc-4851-b878-b25241d19a46\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") "
Sep 09 03:18:50.239504 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.239431    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-oauth-serving-cert\") pod \"861c585a-cfcc-4851-b878-b25241d19a46\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") "
Sep 09 03:18:50.239685 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.239566    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqclr\" (UniqueName: \"kubernetes.io/projected/861c585a-cfcc-4851-b878-b25241d19a46-kube-api-access-xqclr\") pod \"861c585a-cfcc-4851-b878-b25241d19a46\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") "
Sep 09 03:18:50.239685 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.239627    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-console-config\") pod \"861c585a-cfcc-4851-b878-b25241d19a46\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") "
Sep 09 03:18:50.239685 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.239645    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-service-ca" (OuterVolumeSpecName: "service-ca") pod "861c585a-cfcc-4851-b878-b25241d19a46" (UID: "861c585a-cfcc-4851-b878-b25241d19a46"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:18:50.239685 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.239673    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-trusted-ca-bundle\") pod \"861c585a-cfcc-4851-b878-b25241d19a46\" (UID: \"861c585a-cfcc-4851-b878-b25241d19a46\") "
Sep 09 03:18:50.239871 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.239803    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "861c585a-cfcc-4851-b878-b25241d19a46" (UID: "861c585a-cfcc-4851-b878-b25241d19a46"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:18:50.240037 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.239987    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-oauth-serving-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:50.240037 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.240011    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:50.240037 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.240036    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-console-config" (OuterVolumeSpecName: "console-config") pod "861c585a-cfcc-4851-b878-b25241d19a46" (UID: "861c585a-cfcc-4851-b878-b25241d19a46"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:18:50.240243 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.240126    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "861c585a-cfcc-4851-b878-b25241d19a46" (UID: "861c585a-cfcc-4851-b878-b25241d19a46"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:18:50.241910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.241882    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/861c585a-cfcc-4851-b878-b25241d19a46-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "861c585a-cfcc-4851-b878-b25241d19a46" (UID: "861c585a-cfcc-4851-b878-b25241d19a46"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:18:50.242017 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.241904    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/861c585a-cfcc-4851-b878-b25241d19a46-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "861c585a-cfcc-4851-b878-b25241d19a46" (UID: "861c585a-cfcc-4851-b878-b25241d19a46"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:18:50.242017 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.241952    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861c585a-cfcc-4851-b878-b25241d19a46-kube-api-access-xqclr" (OuterVolumeSpecName: "kube-api-access-xqclr") pod "861c585a-cfcc-4851-b878-b25241d19a46" (UID: "861c585a-cfcc-4851-b878-b25241d19a46"). InnerVolumeSpecName "kube-api-access-xqclr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:18:50.341261 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.341226    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/861c585a-cfcc-4851-b878-b25241d19a46-console-serving-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:50.341261 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.341258    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqclr\" (UniqueName: \"kubernetes.io/projected/861c585a-cfcc-4851-b878-b25241d19a46-kube-api-access-xqclr\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:50.341459 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.341273    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-console-config\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:50.341459 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.341285    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/861c585a-cfcc-4851-b878-b25241d19a46-trusted-ca-bundle\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:50.341459 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.341297    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/861c585a-cfcc-4851-b878-b25241d19a46-console-oauth-config\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:18:50.673806 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.673766    2560 generic.go:356] "Generic (PLEG): container finished" podID="861c585a-cfcc-4851-b878-b25241d19a46" containerID="1b3f3218f3fae66ab1d91cbc1351824d917168ca6edf4dd25b3feca9faa88562" exitCode=0
Sep 09 03:18:50.673806 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.673804    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59fdb9cc59-9dkqj"
Sep 09 03:18:50.674013 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.673830    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59fdb9cc59-9dkqj" event={"ID":"861c585a-cfcc-4851-b878-b25241d19a46","Type":"ContainerDied","Data":"1b3f3218f3fae66ab1d91cbc1351824d917168ca6edf4dd25b3feca9faa88562"}
Sep 09 03:18:50.674013 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.673883    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59fdb9cc59-9dkqj" event={"ID":"861c585a-cfcc-4851-b878-b25241d19a46","Type":"ContainerDied","Data":"27f0e9af491b7e915fcbe84671d9c79a414a57036464c0d21002a940fbdccfa6"}
Sep 09 03:18:50.674013 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.673916    2560 scope.go:117] "RemoveContainer" containerID="1b3f3218f3fae66ab1d91cbc1351824d917168ca6edf4dd25b3feca9faa88562"
Sep 09 03:18:50.683308 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.683292    2560 scope.go:117] "RemoveContainer" containerID="1b3f3218f3fae66ab1d91cbc1351824d917168ca6edf4dd25b3feca9faa88562"
Sep 09 03:18:50.683667 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:18:50.683648    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3f3218f3fae66ab1d91cbc1351824d917168ca6edf4dd25b3feca9faa88562\": container with ID starting with 1b3f3218f3fae66ab1d91cbc1351824d917168ca6edf4dd25b3feca9faa88562 not found: ID does not exist" containerID="1b3f3218f3fae66ab1d91cbc1351824d917168ca6edf4dd25b3feca9faa88562"
Sep 09 03:18:50.683719 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.683678    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3f3218f3fae66ab1d91cbc1351824d917168ca6edf4dd25b3feca9faa88562"} err="failed to get container status \"1b3f3218f3fae66ab1d91cbc1351824d917168ca6edf4dd25b3feca9faa88562\": rpc error: code = NotFound desc = could not find container \"1b3f3218f3fae66ab1d91cbc1351824d917168ca6edf4dd25b3feca9faa88562\": container with ID starting with 1b3f3218f3fae66ab1d91cbc1351824d917168ca6edf4dd25b3feca9faa88562 not found: ID does not exist"
Sep 09 03:18:50.702119 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.699638    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59fdb9cc59-9dkqj"]
Sep 09 03:18:50.702830 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:50.702806    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59fdb9cc59-9dkqj"]
Sep 09 03:18:51.361537 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:51.361494    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="861c585a-cfcc-4851-b878-b25241d19a46" path="/var/lib/kubelet/pods/861c585a-cfcc-4851-b878-b25241d19a46/volumes"
Sep 09 03:18:52.994044 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:52.994011    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:18:53.695002 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:53.694968    2560 generic.go:356] "Generic (PLEG): container finished" podID="9d8945e0-9f39-4523-b066-5125ad017318" containerID="39ffdf2d8d77f4ca6802ff8c4504e1a4c86f4fcc22dbda7f59a5c306b9d418c8" exitCode=0
Sep 09 03:18:53.695165 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:53.695040    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn" event={"ID":"9d8945e0-9f39-4523-b066-5125ad017318","Type":"ContainerDied","Data":"39ffdf2d8d77f4ca6802ff8c4504e1a4c86f4fcc22dbda7f59a5c306b9d418c8"}
Sep 09 03:18:53.695393 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:53.695380    2560 scope.go:117] "RemoveContainer" containerID="39ffdf2d8d77f4ca6802ff8c4504e1a4c86f4fcc22dbda7f59a5c306b9d418c8"
Sep 09 03:18:54.701660 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:18:54.701624    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dhtsn" event={"ID":"9d8945e0-9f39-4523-b066-5125ad017318","Type":"ContainerStarted","Data":"8b410cbbbf8d2a55e9860edad7e9e7346aaf75b3bca05660b4a854e6bc5f45c8"}
Sep 09 03:19:00.662466 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:00.662352    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f5d7f97f6-p8x5h" podUID="b1308509-b56e-4f2d-a21a-2687d7f649e1" containerName="console" containerID="cri-o://27bf5b239f39cb539cbcbcfbf0ed3a97625f16c1f75b5451d9b85b452c5c6745" gracePeriod=15
Sep 09 03:19:00.947942 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:00.947910    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:19:01.043922 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.043894    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-service-ca\") pod \"b1308509-b56e-4f2d-a21a-2687d7f649e1\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") "
Sep 09 03:19:01.044089 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.043941    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-console-config\") pod \"b1308509-b56e-4f2d-a21a-2687d7f649e1\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") "
Sep 09 03:19:01.044089 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.043987    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-oauth-serving-cert\") pod \"b1308509-b56e-4f2d-a21a-2687d7f649e1\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") "
Sep 09 03:19:01.044089 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.044041    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ww7g\" (UniqueName: \"kubernetes.io/projected/b1308509-b56e-4f2d-a21a-2687d7f649e1-kube-api-access-7ww7g\") pod \"b1308509-b56e-4f2d-a21a-2687d7f649e1\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") "
Sep 09 03:19:01.044089 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.044064    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1308509-b56e-4f2d-a21a-2687d7f649e1-console-serving-cert\") pod \"b1308509-b56e-4f2d-a21a-2687d7f649e1\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") "
Sep 09 03:19:01.044089 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.044083    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1308509-b56e-4f2d-a21a-2687d7f649e1-console-oauth-config\") pod \"b1308509-b56e-4f2d-a21a-2687d7f649e1\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") "
Sep 09 03:19:01.044394 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.044101    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-trusted-ca-bundle\") pod \"b1308509-b56e-4f2d-a21a-2687d7f649e1\" (UID: \"b1308509-b56e-4f2d-a21a-2687d7f649e1\") "
Sep 09 03:19:01.044394 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.044346    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-service-ca" (OuterVolumeSpecName: "service-ca") pod "b1308509-b56e-4f2d-a21a-2687d7f649e1" (UID: "b1308509-b56e-4f2d-a21a-2687d7f649e1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:19:01.044394 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.044361    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b1308509-b56e-4f2d-a21a-2687d7f649e1" (UID: "b1308509-b56e-4f2d-a21a-2687d7f649e1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:19:01.044533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.044427    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-console-config" (OuterVolumeSpecName: "console-config") pod "b1308509-b56e-4f2d-a21a-2687d7f649e1" (UID: "b1308509-b56e-4f2d-a21a-2687d7f649e1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:19:01.044674 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.044650    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b1308509-b56e-4f2d-a21a-2687d7f649e1" (UID: "b1308509-b56e-4f2d-a21a-2687d7f649e1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:19:01.046212 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.046191    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1308509-b56e-4f2d-a21a-2687d7f649e1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b1308509-b56e-4f2d-a21a-2687d7f649e1" (UID: "b1308509-b56e-4f2d-a21a-2687d7f649e1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:19:01.046642 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.046624    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1308509-b56e-4f2d-a21a-2687d7f649e1-kube-api-access-7ww7g" (OuterVolumeSpecName: "kube-api-access-7ww7g") pod "b1308509-b56e-4f2d-a21a-2687d7f649e1" (UID: "b1308509-b56e-4f2d-a21a-2687d7f649e1"). InnerVolumeSpecName "kube-api-access-7ww7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:19:01.046704 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.046623    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1308509-b56e-4f2d-a21a-2687d7f649e1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b1308509-b56e-4f2d-a21a-2687d7f649e1" (UID: "b1308509-b56e-4f2d-a21a-2687d7f649e1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:19:01.145761 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.145719    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ww7g\" (UniqueName: \"kubernetes.io/projected/b1308509-b56e-4f2d-a21a-2687d7f649e1-kube-api-access-7ww7g\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:19:01.145761 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.145751    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1308509-b56e-4f2d-a21a-2687d7f649e1-console-serving-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:19:01.145761 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.145762    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1308509-b56e-4f2d-a21a-2687d7f649e1-console-oauth-config\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:19:01.145761 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.145770    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-trusted-ca-bundle\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:19:01.146020 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.145780    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:19:01.146020 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.145788    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-console-config\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:19:01.146020 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.145796    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1308509-b56e-4f2d-a21a-2687d7f649e1-oauth-serving-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:19:01.734336 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.734243    2560 generic.go:356] "Generic (PLEG): container finished" podID="b1308509-b56e-4f2d-a21a-2687d7f649e1" containerID="27bf5b239f39cb539cbcbcfbf0ed3a97625f16c1f75b5451d9b85b452c5c6745" exitCode=0
Sep 09 03:19:01.734336 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.734306    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5d7f97f6-p8x5h"
Sep 09 03:19:01.734336 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.734325    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5d7f97f6-p8x5h" event={"ID":"b1308509-b56e-4f2d-a21a-2687d7f649e1","Type":"ContainerDied","Data":"27bf5b239f39cb539cbcbcfbf0ed3a97625f16c1f75b5451d9b85b452c5c6745"}
Sep 09 03:19:01.734844 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.734411    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5d7f97f6-p8x5h" event={"ID":"b1308509-b56e-4f2d-a21a-2687d7f649e1","Type":"ContainerDied","Data":"78484aeba7d5b05f27d69fad2affabe1d68134f011cb84537ceeeed2e3c10b81"}
Sep 09 03:19:01.734844 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.734437    2560 scope.go:117] "RemoveContainer" containerID="27bf5b239f39cb539cbcbcfbf0ed3a97625f16c1f75b5451d9b85b452c5c6745"
Sep 09 03:19:01.743001 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.742981    2560 scope.go:117] "RemoveContainer" containerID="27bf5b239f39cb539cbcbcfbf0ed3a97625f16c1f75b5451d9b85b452c5c6745"
Sep 09 03:19:01.743326 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:19:01.743294    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27bf5b239f39cb539cbcbcfbf0ed3a97625f16c1f75b5451d9b85b452c5c6745\": container with ID starting with 27bf5b239f39cb539cbcbcfbf0ed3a97625f16c1f75b5451d9b85b452c5c6745 not found: ID does not exist" containerID="27bf5b239f39cb539cbcbcfbf0ed3a97625f16c1f75b5451d9b85b452c5c6745"
Sep 09 03:19:01.743383 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.743325    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27bf5b239f39cb539cbcbcfbf0ed3a97625f16c1f75b5451d9b85b452c5c6745"} err="failed to get container status \"27bf5b239f39cb539cbcbcfbf0ed3a97625f16c1f75b5451d9b85b452c5c6745\": rpc error: code = NotFound desc = could not find container \"27bf5b239f39cb539cbcbcfbf0ed3a97625f16c1f75b5451d9b85b452c5c6745\": container with ID starting with 27bf5b239f39cb539cbcbcfbf0ed3a97625f16c1f75b5451d9b85b452c5c6745 not found: ID does not exist"
Sep 09 03:19:01.751464 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.751434    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f5d7f97f6-p8x5h"]
Sep 09 03:19:01.755058 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:01.755032    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f5d7f97f6-p8x5h"]
Sep 09 03:19:03.359134 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:03.359098    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1308509-b56e-4f2d-a21a-2687d7f649e1" path="/var/lib/kubelet/pods/b1308509-b56e-4f2d-a21a-2687d7f649e1/volumes"
Sep 09 03:19:17.385137 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:17.385101    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs\") pod \"network-metrics-daemon-hlmbb\" (UID: \"f8508862-2ffb-4bd8-bb4f-67cd69a78075\") " pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:19:17.387290 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:17.387271    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8508862-2ffb-4bd8-bb4f-67cd69a78075-metrics-certs\") pod \"network-metrics-daemon-hlmbb\" (UID: \"f8508862-2ffb-4bd8-bb4f-67cd69a78075\") " pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:19:17.657501 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:17.657425    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rkcrq\""
Sep 09 03:19:17.665056 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:17.665039    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlmbb"
Sep 09 03:19:17.936677 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:17.936589    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hlmbb"]
Sep 09 03:19:17.940390 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:19:17.940364    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8508862_2ffb_4bd8_bb4f_67cd69a78075.slice/crio-3dbbdc1c428f7c50e3012c4304f111e80ea410c52997fc3695e54bcdf835c0ad WatchSource:0}: Error finding container 3dbbdc1c428f7c50e3012c4304f111e80ea410c52997fc3695e54bcdf835c0ad: Status 404 returned error can't find the container with id 3dbbdc1c428f7c50e3012c4304f111e80ea410c52997fc3695e54bcdf835c0ad
Sep 09 03:19:18.802255 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:18.802213    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hlmbb" event={"ID":"f8508862-2ffb-4bd8-bb4f-67cd69a78075","Type":"ContainerStarted","Data":"3dbbdc1c428f7c50e3012c4304f111e80ea410c52997fc3695e54bcdf835c0ad"}
Sep 09 03:19:19.807504 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:19.807465    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hlmbb" event={"ID":"f8508862-2ffb-4bd8-bb4f-67cd69a78075","Type":"ContainerStarted","Data":"bc98ed8ef58116d67ada37a25ca2056f03f79c719fb1249f01c623528ec417f4"}
Sep 09 03:19:19.807895 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:19.807520    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hlmbb" event={"ID":"f8508862-2ffb-4bd8-bb4f-67cd69a78075","Type":"ContainerStarted","Data":"206fb6dfb388194855c962343088a2dfb5f77c1d8bd42b908427d99014948feb"}
Sep 09 03:19:19.826804 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:19.826759    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hlmbb" podStartSLOduration=374.804528745 podStartE2EDuration="6m15.826744092s" totalImagesPullingTime="1.022215347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:19:19.825301468 +0000 UTC m=+376.962049356" watchObservedRunningTime="2026-09-09 03:19:19.826744092 +0000 UTC m=+376.963491958"
Sep 09 03:19:19.928264 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:19.928230    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:20.029301 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:20.029270    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:21.359085 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:21.359051    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:19:21.377253 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:21.377229    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:19:21.740400 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:21.740313    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:21.831385 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:21.831359    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 03:19:21.838952 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:21.838925    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:26.098000 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:26.097961    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:26.198971 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:26.198939    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:30.910772 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:30.910736    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:31.002704 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:31.002674    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:41.045569 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.045534    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c7bcd5696-r6zk4"]
Sep 09 03:19:41.046613 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.046567    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="861c585a-cfcc-4851-b878-b25241d19a46" containerName="console"
Sep 09 03:19:41.046764 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.046621    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="861c585a-cfcc-4851-b878-b25241d19a46" containerName="console"
Sep 09 03:19:41.046764 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.046641    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87fed466-b71f-4bc5-b0e0-447ebecbaacb" containerName="registry"
Sep 09 03:19:41.046764 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.046649    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="87fed466-b71f-4bc5-b0e0-447ebecbaacb" containerName="registry"
Sep 09 03:19:41.046764 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.046669    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51a904dd-aa8a-4689-9635-e31d15b97dd0" containerName="console"
Sep 09 03:19:41.046764 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.046678    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="51a904dd-aa8a-4689-9635-e31d15b97dd0" containerName="console"
Sep 09 03:19:41.046764 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.046699    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1308509-b56e-4f2d-a21a-2687d7f649e1" containerName="console"
Sep 09 03:19:41.046764 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.046708    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1308509-b56e-4f2d-a21a-2687d7f649e1" containerName="console"
Sep 09 03:19:41.047127 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.046798    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="87fed466-b71f-4bc5-b0e0-447ebecbaacb" containerName="registry"
Sep 09 03:19:41.047127 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.046814    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="51a904dd-aa8a-4689-9635-e31d15b97dd0" containerName="console"
Sep 09 03:19:41.047127 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.046824    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1308509-b56e-4f2d-a21a-2687d7f649e1" containerName="console"
Sep 09 03:19:41.047127 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.046832    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="861c585a-cfcc-4851-b878-b25241d19a46" containerName="console"
Sep 09 03:19:41.050990 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.050971    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.064973 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.064949    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c7bcd5696-r6zk4"]
Sep 09 03:19:41.130844 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.130810    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:41.179217 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.179190    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-trusted-ca-bundle\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.179217 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.179220    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-console-serving-cert\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.179405 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.179265    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p97g7\" (UniqueName: \"kubernetes.io/projected/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-kube-api-access-p97g7\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.179405 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.179351    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-oauth-serving-cert\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.179405 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.179387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-service-ca\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.179557 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.179410    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-console-config\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.179557 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.179479    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-console-oauth-config\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.192036 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.192009    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:41.280487 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.280445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-trusted-ca-bundle\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.280487 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.280488    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-console-serving-cert\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.280747 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.280530    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p97g7\" (UniqueName: \"kubernetes.io/projected/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-kube-api-access-p97g7\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.280747 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.280565    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-oauth-serving-cert\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.280747 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.280590    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-service-ca\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.280747 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.280640    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-console-config\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.280747 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.280674    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-console-oauth-config\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.281404 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.281377    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-console-config\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.281404 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.281390    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-service-ca\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.281553 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.281395    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-oauth-serving-cert\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.281553 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.281440    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-trusted-ca-bundle\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.283035 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.283012    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-console-serving-cert\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.283114 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.283041    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-console-oauth-config\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.289608 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.289571    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p97g7\" (UniqueName: \"kubernetes.io/projected/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-kube-api-access-p97g7\") pod \"console-6c7bcd5696-r6zk4\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") " pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.360651 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.360619    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:41.634905 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.634878    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c7bcd5696-r6zk4"]
Sep 09 03:19:41.636981 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:19:41.636948    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a78c4e9_bac0_4b29_be5d_a644ffe7261c.slice/crio-d24f5cd78fd630c8cf6dc65b39140abea0cc0bdc25c4a7804bbe346e0272fb88 WatchSource:0}: Error finding container d24f5cd78fd630c8cf6dc65b39140abea0cc0bdc25c4a7804bbe346e0272fb88: Status 404 returned error can't find the container with id d24f5cd78fd630c8cf6dc65b39140abea0cc0bdc25c4a7804bbe346e0272fb88
Sep 09 03:19:41.897964 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.897888    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7bcd5696-r6zk4" event={"ID":"9a78c4e9-bac0-4b29-be5d-a644ffe7261c","Type":"ContainerStarted","Data":"1e68e38020d86ecb6fe0e450affb647614780b6af82f507d7f03e6e15dc03bda"}
Sep 09 03:19:41.897964 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.897940    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7bcd5696-r6zk4" event={"ID":"9a78c4e9-bac0-4b29-be5d-a644ffe7261c","Type":"ContainerStarted","Data":"d24f5cd78fd630c8cf6dc65b39140abea0cc0bdc25c4a7804bbe346e0272fb88"}
Sep 09 03:19:41.917913 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:41.917868    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c7bcd5696-r6zk4" podStartSLOduration=0.917853489 podStartE2EDuration="917.853489ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:19:41.916957509 +0000 UTC m=+399.053705376" watchObservedRunningTime="2026-09-09 03:19:41.917853489 +0000 UTC m=+399.054601354"
Sep 09 03:19:42.886583 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:42.886550    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:42.987889 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:42.987853    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:48.955159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:48.955128    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:49.056624 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:49.056576    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:49.130329 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:49.130296    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:51.361745 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:51.361709    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:51.362198 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:51.361772    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:51.366776 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:51.366749    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:51.951693 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:51.951664    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:19:52.006666 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:52.006635    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-776df47b56-lpj2g"]
Sep 09 03:19:52.116086 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:52.116051    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:52.222887 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:52.222804    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:19:52.345922 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:19:52.345886    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:09.652534 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:09.652498    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:09.753879 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:09.753845    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:17.032949 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.032892    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-776df47b56-lpj2g" podUID="117902b2-b67c-48e2-88c0-ac7a2560d122" containerName="console" containerID="cri-o://ae94cc7081fedea87737927ec3d07c2e2feca1d79d039a8e3e08e6b56c84f1db" gracePeriod=15
Sep 09 03:20:17.263256 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.263225    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:20:17.287821 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.287751    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-service-ca\") pod \"117902b2-b67c-48e2-88c0-ac7a2560d122\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") "
Sep 09 03:20:17.287821 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.287804    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/117902b2-b67c-48e2-88c0-ac7a2560d122-console-serving-cert\") pod \"117902b2-b67c-48e2-88c0-ac7a2560d122\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") "
Sep 09 03:20:17.288007 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.287840    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-console-config\") pod \"117902b2-b67c-48e2-88c0-ac7a2560d122\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") "
Sep 09 03:20:17.288007 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.287893    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/117902b2-b67c-48e2-88c0-ac7a2560d122-console-oauth-config\") pod \"117902b2-b67c-48e2-88c0-ac7a2560d122\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") "
Sep 09 03:20:17.288007 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.287962    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-trusted-ca-bundle\") pod \"117902b2-b67c-48e2-88c0-ac7a2560d122\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") "
Sep 09 03:20:17.288163 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.288007    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvxv4\" (UniqueName: \"kubernetes.io/projected/117902b2-b67c-48e2-88c0-ac7a2560d122-kube-api-access-jvxv4\") pod \"117902b2-b67c-48e2-88c0-ac7a2560d122\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") "
Sep 09 03:20:17.288163 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.288052    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-oauth-serving-cert\") pod \"117902b2-b67c-48e2-88c0-ac7a2560d122\" (UID: \"117902b2-b67c-48e2-88c0-ac7a2560d122\") "
Sep 09 03:20:17.288268 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.288213    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-service-ca" (OuterVolumeSpecName: "service-ca") pod "117902b2-b67c-48e2-88c0-ac7a2560d122" (UID: "117902b2-b67c-48e2-88c0-ac7a2560d122"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:20:17.288268 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.288255    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-console-config" (OuterVolumeSpecName: "console-config") pod "117902b2-b67c-48e2-88c0-ac7a2560d122" (UID: "117902b2-b67c-48e2-88c0-ac7a2560d122"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:20:17.288374 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.288350    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-console-config\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:20:17.288374 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.288367    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:20:17.288572 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.288550    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "117902b2-b67c-48e2-88c0-ac7a2560d122" (UID: "117902b2-b67c-48e2-88c0-ac7a2560d122"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:20:17.288716 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.288689    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "117902b2-b67c-48e2-88c0-ac7a2560d122" (UID: "117902b2-b67c-48e2-88c0-ac7a2560d122"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:20:17.290158 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.290131    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/117902b2-b67c-48e2-88c0-ac7a2560d122-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "117902b2-b67c-48e2-88c0-ac7a2560d122" (UID: "117902b2-b67c-48e2-88c0-ac7a2560d122"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:20:17.290259 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.290240    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/117902b2-b67c-48e2-88c0-ac7a2560d122-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "117902b2-b67c-48e2-88c0-ac7a2560d122" (UID: "117902b2-b67c-48e2-88c0-ac7a2560d122"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:20:17.291006 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.290986    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117902b2-b67c-48e2-88c0-ac7a2560d122-kube-api-access-jvxv4" (OuterVolumeSpecName: "kube-api-access-jvxv4") pod "117902b2-b67c-48e2-88c0-ac7a2560d122" (UID: "117902b2-b67c-48e2-88c0-ac7a2560d122"). InnerVolumeSpecName "kube-api-access-jvxv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:20:17.389636 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.389587    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-trusted-ca-bundle\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:20:17.389636 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.389632    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvxv4\" (UniqueName: \"kubernetes.io/projected/117902b2-b67c-48e2-88c0-ac7a2560d122-kube-api-access-jvxv4\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:20:17.389805 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.389645    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/117902b2-b67c-48e2-88c0-ac7a2560d122-oauth-serving-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:20:17.389805 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.389658    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/117902b2-b67c-48e2-88c0-ac7a2560d122-console-serving-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:20:17.389805 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:17.389670    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/117902b2-b67c-48e2-88c0-ac7a2560d122-console-oauth-config\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:20:18.061774 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:18.061738    2560 generic.go:356] "Generic (PLEG): container finished" podID="117902b2-b67c-48e2-88c0-ac7a2560d122" containerID="ae94cc7081fedea87737927ec3d07c2e2feca1d79d039a8e3e08e6b56c84f1db" exitCode=0
Sep 09 03:20:18.062159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:18.061786    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776df47b56-lpj2g"
Sep 09 03:20:18.062159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:18.061824    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776df47b56-lpj2g" event={"ID":"117902b2-b67c-48e2-88c0-ac7a2560d122","Type":"ContainerDied","Data":"ae94cc7081fedea87737927ec3d07c2e2feca1d79d039a8e3e08e6b56c84f1db"}
Sep 09 03:20:18.062159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:18.061877    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776df47b56-lpj2g" event={"ID":"117902b2-b67c-48e2-88c0-ac7a2560d122","Type":"ContainerDied","Data":"8e5bde5a8ff48e75beb2424a6a11006e614b92ef83918a13da338bc8548c4780"}
Sep 09 03:20:18.062159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:18.061896    2560 scope.go:117] "RemoveContainer" containerID="ae94cc7081fedea87737927ec3d07c2e2feca1d79d039a8e3e08e6b56c84f1db"
Sep 09 03:20:18.070096 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:18.070080    2560 scope.go:117] "RemoveContainer" containerID="ae94cc7081fedea87737927ec3d07c2e2feca1d79d039a8e3e08e6b56c84f1db"
Sep 09 03:20:18.070367 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:20:18.070347    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae94cc7081fedea87737927ec3d07c2e2feca1d79d039a8e3e08e6b56c84f1db\": container with ID starting with ae94cc7081fedea87737927ec3d07c2e2feca1d79d039a8e3e08e6b56c84f1db not found: ID does not exist" containerID="ae94cc7081fedea87737927ec3d07c2e2feca1d79d039a8e3e08e6b56c84f1db"
Sep 09 03:20:18.070418 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:18.070375    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae94cc7081fedea87737927ec3d07c2e2feca1d79d039a8e3e08e6b56c84f1db"} err="failed to get container status \"ae94cc7081fedea87737927ec3d07c2e2feca1d79d039a8e3e08e6b56c84f1db\": rpc error: code = NotFound desc = could not find container \"ae94cc7081fedea87737927ec3d07c2e2feca1d79d039a8e3e08e6b56c84f1db\": container with ID starting with ae94cc7081fedea87737927ec3d07c2e2feca1d79d039a8e3e08e6b56c84f1db not found: ID does not exist"
Sep 09 03:20:18.078472 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:18.078449    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-776df47b56-lpj2g"]
Sep 09 03:20:18.084279 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:18.084258    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-776df47b56-lpj2g"]
Sep 09 03:20:19.358710 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:19.358677    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="117902b2-b67c-48e2-88c0-ac7a2560d122" path="/var/lib/kubelet/pods/117902b2-b67c-48e2-88c0-ac7a2560d122/volumes"
Sep 09 03:20:31.595882 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:31.595804    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:31.703503 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:31.703471    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:53.492911 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:53.492868    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:54.345951 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:54.345911    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:57.127163 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:57.127126    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:57.223822 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:57.223787    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:59.123334 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:59.123300    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:59.222226 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:59.222190    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:59.530789 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:59.530702    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:20:59.649527 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:20:59.649493    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:01.158273 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:01.158239    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:01.523062 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:01.522981    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:04.106753 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:04.106715    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:04.208326 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:04.208286    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:09.939520 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:09.939489    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:12.297062 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:12.297029    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:12.395134 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:12.395107    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:13.567542 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:13.567507    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vkwbv"]
Sep 09 03:21:13.567946 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:13.567849    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="117902b2-b67c-48e2-88c0-ac7a2560d122" containerName="console"
Sep 09 03:21:13.567946 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:13.567860    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="117902b2-b67c-48e2-88c0-ac7a2560d122" containerName="console"
Sep 09 03:21:13.567946 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:13.567922    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="117902b2-b67c-48e2-88c0-ac7a2560d122" containerName="console"
Sep 09 03:21:13.571991 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:13.571975    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vkwbv"
Sep 09 03:21:13.574706 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:13.574687    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:21:13.575637 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:13.575617    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-29tkn\""
Sep 09 03:21:13.575746 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:13.575669    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 03:21:13.580370 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:13.580348    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vkwbv"]
Sep 09 03:21:13.660585 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:13.660549    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrvkx\" (UniqueName: \"kubernetes.io/projected/819bbf51-7f42-46c3-8263-e2aefdc7cc3c-kube-api-access-lrvkx\") pod \"openshift-lws-operator-fd8ccff4c-vkwbv\" (UID: \"819bbf51-7f42-46c3-8263-e2aefdc7cc3c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vkwbv"
Sep 09 03:21:13.660778 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:13.660662    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/819bbf51-7f42-46c3-8263-e2aefdc7cc3c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vkwbv\" (UID: \"819bbf51-7f42-46c3-8263-e2aefdc7cc3c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vkwbv"
Sep 09 03:21:13.761810 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:13.761777    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/819bbf51-7f42-46c3-8263-e2aefdc7cc3c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vkwbv\" (UID: \"819bbf51-7f42-46c3-8263-e2aefdc7cc3c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vkwbv"
Sep 09 03:21:13.761975 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:13.761867    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrvkx\" (UniqueName: \"kubernetes.io/projected/819bbf51-7f42-46c3-8263-e2aefdc7cc3c-kube-api-access-lrvkx\") pod \"openshift-lws-operator-fd8ccff4c-vkwbv\" (UID: \"819bbf51-7f42-46c3-8263-e2aefdc7cc3c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vkwbv"
Sep 09 03:21:13.762134 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:13.762112    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/819bbf51-7f42-46c3-8263-e2aefdc7cc3c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vkwbv\" (UID: \"819bbf51-7f42-46c3-8263-e2aefdc7cc3c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vkwbv"
Sep 09 03:21:13.770061 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:13.770034    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrvkx\" (UniqueName: \"kubernetes.io/projected/819bbf51-7f42-46c3-8263-e2aefdc7cc3c-kube-api-access-lrvkx\") pod \"openshift-lws-operator-fd8ccff4c-vkwbv\" (UID: \"819bbf51-7f42-46c3-8263-e2aefdc7cc3c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vkwbv"
Sep 09 03:21:13.891385 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:13.891359    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vkwbv"
Sep 09 03:21:14.163000 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:14.162973    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vkwbv"]
Sep 09 03:21:14.164654 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:21:14.164628    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod819bbf51_7f42_46c3_8263_e2aefdc7cc3c.slice/crio-ecf90dd01276b1cadd78475bdd7540adfd92a9f7347ddfbaa488de3dc6a0569b WatchSource:0}: Error finding container ecf90dd01276b1cadd78475bdd7540adfd92a9f7347ddfbaa488de3dc6a0569b: Status 404 returned error can't find the container with id ecf90dd01276b1cadd78475bdd7540adfd92a9f7347ddfbaa488de3dc6a0569b
Sep 09 03:21:14.297386 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:14.297352    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vkwbv" event={"ID":"819bbf51-7f42-46c3-8263-e2aefdc7cc3c","Type":"ContainerStarted","Data":"ecf90dd01276b1cadd78475bdd7540adfd92a9f7347ddfbaa488de3dc6a0569b"}
Sep 09 03:21:17.314559 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:17.314514    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vkwbv" event={"ID":"819bbf51-7f42-46c3-8263-e2aefdc7cc3c","Type":"ContainerStarted","Data":"553372479bd7b711c689ddb6a9e4acda5a76010b26034ff8e7d357d035c2f058"}
Sep 09 03:21:17.333208 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:17.333145    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vkwbv" podStartSLOduration=1.833331987 podStartE2EDuration="4.333123942s" totalImagesPullingTime="2.499791955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:21:17.330930923 +0000 UTC m=+494.467678779" watchObservedRunningTime="2026-09-09 03:21:17.333123942 +0000 UTC m=+494.469871808"
Sep 09 03:21:18.101267 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:18.101232    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:18.228511 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:18.228471    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:20.030514 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:20.030476    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:23.644850 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:23.644816    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:23.744087 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:23.744037    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:33.660409 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:33.660374    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:33.758955 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:33.758892    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:36.145995 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:36.145963    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:36.237825 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:36.237791    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:36.804956 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:36.804928    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:37.091571 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.091491    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:37.215781 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.215750    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:37.267208 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.267175    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"]
Sep 09 03:21:37.270534 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.270516    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:37.273696 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.273666    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 03:21:37.273696 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.273679    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 03:21:37.273855 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.273723    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jcqpg\""
Sep 09 03:21:37.273855 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.273679    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 03:21:37.274024 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.274010    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 03:21:37.297498 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.297469    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"]
Sep 09 03:21:37.364738 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.364711    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624-apiservice-cert\") pod \"opendatahub-operator-controller-manager-ff49c877-vsrxv\" (UID: \"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624\") " pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:37.364897 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.364779    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624-tmp\") pod \"opendatahub-operator-controller-manager-ff49c877-vsrxv\" (UID: \"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624\") " pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:37.364897 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.364826    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624-webhook-cert\") pod \"opendatahub-operator-controller-manager-ff49c877-vsrxv\" (UID: \"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624\") " pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:37.364897 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.364868    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624-manifests\") pod \"opendatahub-operator-controller-manager-ff49c877-vsrxv\" (UID: \"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624\") " pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:37.364897 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.364894    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tplm2\" (UniqueName: \"kubernetes.io/projected/4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624-kube-api-access-tplm2\") pod \"opendatahub-operator-controller-manager-ff49c877-vsrxv\" (UID: \"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624\") " pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:37.391559 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.391531    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:37.466336 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.466303    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624-tmp\") pod \"opendatahub-operator-controller-manager-ff49c877-vsrxv\" (UID: \"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624\") " pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:37.466532 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.466347    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624-webhook-cert\") pod \"opendatahub-operator-controller-manager-ff49c877-vsrxv\" (UID: \"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624\") " pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:37.466532 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.466385    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624-manifests\") pod \"opendatahub-operator-controller-manager-ff49c877-vsrxv\" (UID: \"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624\") " pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:37.466532 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.466424    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tplm2\" (UniqueName: \"kubernetes.io/projected/4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624-kube-api-access-tplm2\") pod \"opendatahub-operator-controller-manager-ff49c877-vsrxv\" (UID: \"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624\") " pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:37.466532 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.466465    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624-apiservice-cert\") pod \"opendatahub-operator-controller-manager-ff49c877-vsrxv\" (UID: \"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624\") " pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:37.466871 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.466848    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624-manifests\") pod \"opendatahub-operator-controller-manager-ff49c877-vsrxv\" (UID: \"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624\") " pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:37.468764 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.468735    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624-tmp\") pod \"opendatahub-operator-controller-manager-ff49c877-vsrxv\" (UID: \"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624\") " pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:37.468859 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.468809    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624-apiservice-cert\") pod \"opendatahub-operator-controller-manager-ff49c877-vsrxv\" (UID: \"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624\") " pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:37.468859 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.468842    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624-webhook-cert\") pod \"opendatahub-operator-controller-manager-ff49c877-vsrxv\" (UID: \"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624\") " pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:37.476252 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.476227    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tplm2\" (UniqueName: \"kubernetes.io/projected/4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624-kube-api-access-tplm2\") pod \"opendatahub-operator-controller-manager-ff49c877-vsrxv\" (UID: \"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624\") " pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:37.500727 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.500702    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:37.581324 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.581289    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:37.616693 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.616576    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:37.741979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.741948    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:37.843181 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.843153    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:37.862329 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.862306    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"]
Sep 09 03:21:37.864737 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:21:37.864713    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cfb4c1a_4eb4_4dbb_ad2c_b4182f555624.slice/crio-5c9b345b3178c236a9712f87ede72098cecef9660a3a57696fd8b72ec603e296 WatchSource:0}: Error finding container 5c9b345b3178c236a9712f87ede72098cecef9660a3a57696fd8b72ec603e296: Status 404 returned error can't find the container with id 5c9b345b3178c236a9712f87ede72098cecef9660a3a57696fd8b72ec603e296
Sep 09 03:21:37.942152 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:37.942116    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:38.415580 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:38.415547    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv" event={"ID":"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624","Type":"ContainerStarted","Data":"5c9b345b3178c236a9712f87ede72098cecef9660a3a57696fd8b72ec603e296"}
Sep 09 03:21:41.431320 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:41.431284    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv" event={"ID":"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624","Type":"ContainerStarted","Data":"e4d1bcd04886683d2cf2ccf6f35739115ad91cccdec1819a52495644a6cf4c8d"}
Sep 09 03:21:42.437304 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:42.437265    2560 generic.go:356] "Generic (PLEG): container finished" podID="4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624" containerID="e4d1bcd04886683d2cf2ccf6f35739115ad91cccdec1819a52495644a6cf4c8d" exitCode=0
Sep 09 03:21:42.437689 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:42.437308    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv" event={"ID":"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624","Type":"ContainerDied","Data":"e4d1bcd04886683d2cf2ccf6f35739115ad91cccdec1819a52495644a6cf4c8d"}
Sep 09 03:21:43.446681 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:43.446644    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv" event={"ID":"4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624","Type":"ContainerStarted","Data":"20a6b0762fd3ad0f1297ea0a632f80e9813099d26b04556f55c5c0ecb7d322c3"}
Sep 09 03:21:43.447050 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:43.446822    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:43.474863 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:43.474789    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv" podStartSLOduration=2.141966892 podStartE2EDuration="6.474772805s" totalImagesPullingTime="3.274700822s" totalInitContainerRuntime="1.058105091s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:21:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:21:43.473894226 +0000 UTC m=+520.610642094" watchObservedRunningTime="2026-09-09 03:21:43.474772805 +0000 UTC m=+520.611520733"
Sep 09 03:21:48.972181 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:48.972117    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:49.085993 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:49.085961    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:54.453402 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:54.453368    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-ff49c877-vsrxv"
Sep 09 03:21:54.862435 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:54.862396    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:54.976204 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:54.976175    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:54.976985 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:54.976962    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"]
Sep 09 03:21:54.986252 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:54.986231    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"
Sep 09 03:21:54.990275 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:54.990251    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 03:21:54.990378 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:54.990348    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 03:21:54.990378 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:54.990363    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 03:21:54.990567 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:54.990500    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-r7mfk\""
Sep 09 03:21:54.993490 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:54.993123    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"]
Sep 09 03:21:55.017863 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:55.017831    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b-cert\") pod \"lws-controller-manager-6f5d5c9c64-7pmsk\" (UID: \"0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b\") " pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"
Sep 09 03:21:55.018143 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:55.018122    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b-metrics-cert\") pod \"lws-controller-manager-6f5d5c9c64-7pmsk\" (UID: \"0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b\") " pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"
Sep 09 03:21:55.018298 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:55.018285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b-manager-config\") pod \"lws-controller-manager-6f5d5c9c64-7pmsk\" (UID: \"0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b\") " pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"
Sep 09 03:21:55.018458 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:55.018443    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jfd5\" (UniqueName: \"kubernetes.io/projected/0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b-kube-api-access-7jfd5\") pod \"lws-controller-manager-6f5d5c9c64-7pmsk\" (UID: \"0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b\") " pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"
Sep 09 03:21:55.119760 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:55.119678    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b-metrics-cert\") pod \"lws-controller-manager-6f5d5c9c64-7pmsk\" (UID: \"0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b\") " pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"
Sep 09 03:21:55.119760 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:55.119728    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b-manager-config\") pod \"lws-controller-manager-6f5d5c9c64-7pmsk\" (UID: \"0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b\") " pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"
Sep 09 03:21:55.119948 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:55.119898    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jfd5\" (UniqueName: \"kubernetes.io/projected/0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b-kube-api-access-7jfd5\") pod \"lws-controller-manager-6f5d5c9c64-7pmsk\" (UID: \"0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b\") " pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"
Sep 09 03:21:55.119948 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:55.119936    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b-cert\") pod \"lws-controller-manager-6f5d5c9c64-7pmsk\" (UID: \"0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b\") " pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"
Sep 09 03:21:55.120363 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:55.120342    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b-manager-config\") pod \"lws-controller-manager-6f5d5c9c64-7pmsk\" (UID: \"0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b\") " pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"
Sep 09 03:21:55.122141 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:55.122117    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b-metrics-cert\") pod \"lws-controller-manager-6f5d5c9c64-7pmsk\" (UID: \"0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b\") " pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"
Sep 09 03:21:55.122335 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:55.122315    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b-cert\") pod \"lws-controller-manager-6f5d5c9c64-7pmsk\" (UID: \"0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b\") " pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"
Sep 09 03:21:55.128354 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:55.128329    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jfd5\" (UniqueName: \"kubernetes.io/projected/0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b-kube-api-access-7jfd5\") pod \"lws-controller-manager-6f5d5c9c64-7pmsk\" (UID: \"0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b\") " pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"
Sep 09 03:21:55.296327 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:55.296291    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"
Sep 09 03:21:55.575204 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:55.575180    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"]
Sep 09 03:21:55.577615 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:21:55.577572    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d783a2f_7f25_4bdc_b2b4_0d331b80ad6b.slice/crio-cf36b60eb49c87434978b9ce140cca672209f6dc8d41d870a005cc8586fb5002 WatchSource:0}: Error finding container cf36b60eb49c87434978b9ce140cca672209f6dc8d41d870a005cc8586fb5002: Status 404 returned error can't find the container with id cf36b60eb49c87434978b9ce140cca672209f6dc8d41d870a005cc8586fb5002
Sep 09 03:21:56.506031 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:56.505997    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk" event={"ID":"0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b","Type":"ContainerStarted","Data":"cf36b60eb49c87434978b9ce140cca672209f6dc8d41d870a005cc8586fb5002"}
Sep 09 03:21:56.888827 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:56.888776    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:57.073206 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:57.073165    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:21:58.518218 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:58.518174    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk" event={"ID":"0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b","Type":"ContainerStarted","Data":"341b4d96d8c537021f3ab4ba8337301730a30e4ed1349d76a16e8422ac827fc8"}
Sep 09 03:21:58.518693 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:58.518265    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"
Sep 09 03:21:58.534670 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:21:58.534484    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk" podStartSLOduration=2.597895132 podStartE2EDuration="4.53446301s" totalImagesPullingTime="1.936567878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:21:58.533981493 +0000 UTC m=+535.670729399" watchObservedRunningTime="2026-09-09 03:21:58.53446301 +0000 UTC m=+535.671210927"
Sep 09 03:22:07.743468 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:07.743431    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:07.877441 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:07.877400    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:07.897528 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:07.897499    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58f995d999-7gwbh"]
Sep 09 03:22:07.902353 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:07.902333    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58f995d999-7gwbh"
Sep 09 03:22:07.905551 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:07.905530    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 03:22:07.905715 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:07.905630    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 03:22:07.905778 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:07.905763    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 03:22:07.906019 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:07.906003    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-kwfhb\""
Sep 09 03:22:07.906221 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:07.906201    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 03:22:07.915742 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:07.915718    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58f995d999-7gwbh"]
Sep 09 03:22:08.029568 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:08.029497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97abd95b-8739-4ed4-bfd4-397a25c27de0-tmp\") pod \"kube-auth-proxy-58f995d999-7gwbh\" (UID: \"97abd95b-8739-4ed4-bfd4-397a25c27de0\") " pod="openshift-ingress/kube-auth-proxy-58f995d999-7gwbh"
Sep 09 03:22:08.029568 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:08.029538    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97abd95b-8739-4ed4-bfd4-397a25c27de0-tls-certs\") pod \"kube-auth-proxy-58f995d999-7gwbh\" (UID: \"97abd95b-8739-4ed4-bfd4-397a25c27de0\") " pod="openshift-ingress/kube-auth-proxy-58f995d999-7gwbh"
Sep 09 03:22:08.029568 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:08.029565    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8jdc\" (UniqueName: \"kubernetes.io/projected/97abd95b-8739-4ed4-bfd4-397a25c27de0-kube-api-access-b8jdc\") pod \"kube-auth-proxy-58f995d999-7gwbh\" (UID: \"97abd95b-8739-4ed4-bfd4-397a25c27de0\") " pod="openshift-ingress/kube-auth-proxy-58f995d999-7gwbh"
Sep 09 03:22:08.104695 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:08.104660    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:08.130440 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:08.130415    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8jdc\" (UniqueName: \"kubernetes.io/projected/97abd95b-8739-4ed4-bfd4-397a25c27de0-kube-api-access-b8jdc\") pod \"kube-auth-proxy-58f995d999-7gwbh\" (UID: \"97abd95b-8739-4ed4-bfd4-397a25c27de0\") " pod="openshift-ingress/kube-auth-proxy-58f995d999-7gwbh"
Sep 09 03:22:08.130566 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:08.130517    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97abd95b-8739-4ed4-bfd4-397a25c27de0-tmp\") pod \"kube-auth-proxy-58f995d999-7gwbh\" (UID: \"97abd95b-8739-4ed4-bfd4-397a25c27de0\") " pod="openshift-ingress/kube-auth-proxy-58f995d999-7gwbh"
Sep 09 03:22:08.130566 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:08.130560    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97abd95b-8739-4ed4-bfd4-397a25c27de0-tls-certs\") pod \"kube-auth-proxy-58f995d999-7gwbh\" (UID: \"97abd95b-8739-4ed4-bfd4-397a25c27de0\") " pod="openshift-ingress/kube-auth-proxy-58f995d999-7gwbh"
Sep 09 03:22:08.132720 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:08.132696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97abd95b-8739-4ed4-bfd4-397a25c27de0-tmp\") pod \"kube-auth-proxy-58f995d999-7gwbh\" (UID: \"97abd95b-8739-4ed4-bfd4-397a25c27de0\") " pod="openshift-ingress/kube-auth-proxy-58f995d999-7gwbh"
Sep 09 03:22:08.132880 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:08.132861    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97abd95b-8739-4ed4-bfd4-397a25c27de0-tls-certs\") pod \"kube-auth-proxy-58f995d999-7gwbh\" (UID: \"97abd95b-8739-4ed4-bfd4-397a25c27de0\") " pod="openshift-ingress/kube-auth-proxy-58f995d999-7gwbh"
Sep 09 03:22:08.141117 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:08.141089    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8jdc\" (UniqueName: \"kubernetes.io/projected/97abd95b-8739-4ed4-bfd4-397a25c27de0-kube-api-access-b8jdc\") pod \"kube-auth-proxy-58f995d999-7gwbh\" (UID: \"97abd95b-8739-4ed4-bfd4-397a25c27de0\") " pod="openshift-ingress/kube-auth-proxy-58f995d999-7gwbh"
Sep 09 03:22:08.203438 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:08.203412    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:08.211427 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:08.211409    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58f995d999-7gwbh"
Sep 09 03:22:08.487342 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:08.487321    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58f995d999-7gwbh"]
Sep 09 03:22:08.489643 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:22:08.489617    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97abd95b_8739_4ed4_bfd4_397a25c27de0.slice/crio-14819741a65ad4df3f61ba31a1565df9a9eac0f7776e18520afa892c8aa9c35b WatchSource:0}: Error finding container 14819741a65ad4df3f61ba31a1565df9a9eac0f7776e18520afa892c8aa9c35b: Status 404 returned error can't find the container with id 14819741a65ad4df3f61ba31a1565df9a9eac0f7776e18520afa892c8aa9c35b
Sep 09 03:22:08.491193 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:08.491170    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:08.563854 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:08.563823    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58f995d999-7gwbh" event={"ID":"97abd95b-8739-4ed4-bfd4-397a25c27de0","Type":"ContainerStarted","Data":"14819741a65ad4df3f61ba31a1565df9a9eac0f7776e18520afa892c8aa9c35b"}
Sep 09 03:22:08.741029 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:08.740953    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:09.190358 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:09.190326    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:09.269247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:09.269215    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:09.525169 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:09.525059    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6f5d5c9c64-7pmsk"
Sep 09 03:22:10.251583 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:10.251552    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:11.585193 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:11.585107    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58f995d999-7gwbh" event={"ID":"97abd95b-8739-4ed4-bfd4-397a25c27de0","Type":"ContainerStarted","Data":"0fdebda4d3bd5321fe316e2c6102be48a2400dbd0d0da8d806ef4aca947275b7"}
Sep 09 03:22:11.607451 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:11.607403    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58f995d999-7gwbh" podStartSLOduration=1.949677215 podStartE2EDuration="4.607389243s" totalImagesPullingTime="2.657712028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:22:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:22:11.60569198 +0000 UTC m=+548.742439876" watchObservedRunningTime="2026-09-09 03:22:11.607389243 +0000 UTC m=+548.744137127"
Sep 09 03:22:15.405012 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:15.404979    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:15.530163 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:15.530131    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:16.306643 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:16.306609    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:16.444260 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:16.444231    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:16.585039 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:16.584957    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:18.389840 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:18.389802    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:18.486311 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:18.486275    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:18.640403 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:18.640320    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:18.846822 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:18.846792    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:19.701745 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:19.701715    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:19.802231 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:19.802200    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:22:28.194123 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:22:28.194089    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:09.192059 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:09.192023    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:09.374548 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:09.374519    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:23:09.405067 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:09.405042    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:23:10.634173 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:10.634142    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:25.387228 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:25.387197    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:27.427917 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:27.427883    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:27.604907 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:27.604873    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:32.025164 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:32.025123    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:32.753516 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:32.753474    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:33.556295 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:33.556263    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:35.411983 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:35.411949    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:35.599181 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:35.599147    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:37.033231 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.033197    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69f9b5898b-92slb"]
Sep 09 03:23:37.036781 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.036761    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.045182 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.045159    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:37.061103 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.061081    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69f9b5898b-92slb"]
Sep 09 03:23:37.132464 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.132401    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6780a37d-e6d8-432b-b32c-9082c87315f5-trusted-ca-bundle\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.132643 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.132494    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6780a37d-e6d8-432b-b32c-9082c87315f5-console-serving-cert\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.132643 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.132515    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6780a37d-e6d8-432b-b32c-9082c87315f5-console-config\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.132643 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.132553    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6780a37d-e6d8-432b-b32c-9082c87315f5-service-ca\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.132767 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.132669    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6780a37d-e6d8-432b-b32c-9082c87315f5-oauth-serving-cert\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.132767 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.132714    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqplf\" (UniqueName: \"kubernetes.io/projected/6780a37d-e6d8-432b-b32c-9082c87315f5-kube-api-access-bqplf\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.132767 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.132740    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6780a37d-e6d8-432b-b32c-9082c87315f5-console-oauth-config\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.234161 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.234129    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6780a37d-e6d8-432b-b32c-9082c87315f5-console-serving-cert\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.234161 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.234165    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6780a37d-e6d8-432b-b32c-9082c87315f5-console-config\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.234403 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.234214    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6780a37d-e6d8-432b-b32c-9082c87315f5-service-ca\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.234403 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.234243    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6780a37d-e6d8-432b-b32c-9082c87315f5-oauth-serving-cert\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.234403 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.234269    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqplf\" (UniqueName: \"kubernetes.io/projected/6780a37d-e6d8-432b-b32c-9082c87315f5-kube-api-access-bqplf\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.234403 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.234300    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6780a37d-e6d8-432b-b32c-9082c87315f5-console-oauth-config\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.234403 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.234325    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6780a37d-e6d8-432b-b32c-9082c87315f5-trusted-ca-bundle\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.235107 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.235073    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6780a37d-e6d8-432b-b32c-9082c87315f5-console-config\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.235225 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.235081    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6780a37d-e6d8-432b-b32c-9082c87315f5-oauth-serving-cert\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.235225 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.235142    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6780a37d-e6d8-432b-b32c-9082c87315f5-service-ca\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.235225 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.235200    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6780a37d-e6d8-432b-b32c-9082c87315f5-trusted-ca-bundle\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.236753 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.236718    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6780a37d-e6d8-432b-b32c-9082c87315f5-console-serving-cert\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.236849 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.236768    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6780a37d-e6d8-432b-b32c-9082c87315f5-console-oauth-config\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.285052 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.284993    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqplf\" (UniqueName: \"kubernetes.io/projected/6780a37d-e6d8-432b-b32c-9082c87315f5-kube-api-access-bqplf\") pod \"console-69f9b5898b-92slb\" (UID: \"6780a37d-e6d8-432b-b32c-9082c87315f5\") " pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.345742 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.345702    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:37.418023 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.417993    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:37.641162 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.641138    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69f9b5898b-92slb"]
Sep 09 03:23:37.642991 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:23:37.642965    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6780a37d_e6d8_432b_b32c_9082c87315f5.slice/crio-fbac4fa4740b009b3594ecbce160acac614106a811d7d0fef06f60ab1334e29d WatchSource:0}: Error finding container fbac4fa4740b009b3594ecbce160acac614106a811d7d0fef06f60ab1334e29d: Status 404 returned error can't find the container with id fbac4fa4740b009b3594ecbce160acac614106a811d7d0fef06f60ab1334e29d
Sep 09 03:23:37.978067 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.977977    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f9b5898b-92slb" event={"ID":"6780a37d-e6d8-432b-b32c-9082c87315f5","Type":"ContainerStarted","Data":"88606ee48e1df974390dc8848eee4728728e231e708e750f9713bbefea3556d9"}
Sep 09 03:23:37.978067 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:37.978032    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f9b5898b-92slb" event={"ID":"6780a37d-e6d8-432b-b32c-9082c87315f5","Type":"ContainerStarted","Data":"fbac4fa4740b009b3594ecbce160acac614106a811d7d0fef06f60ab1334e29d"}
Sep 09 03:23:38.019901 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:38.019854    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69f9b5898b-92slb" podStartSLOduration=2.019836402 podStartE2EDuration="2.019836402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:23:38.016088611 +0000 UTC m=+635.152836495" watchObservedRunningTime="2026-09-09 03:23:38.019836402 +0000 UTC m=+635.156584266"
Sep 09 03:23:38.490970 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:38.490935    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:39.253884 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:39.253848    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:41.103936 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:41.103892    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:41.270802 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:41.270747    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:41.782968 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:41.782934    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:43.807001 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:43.806963    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:43.971254 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:43.971218    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:44.910229 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:44.910193    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-ff2jp"]
Sep 09 03:23:44.913789 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:44.913766    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ff2jp"
Sep 09 03:23:44.922146 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:44.922120    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 03:23:44.922276 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:44.922156    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 03:23:44.922276 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:44.922233    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 03:23:44.922391 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:44.922261    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9kbrl\""
Sep 09 03:23:44.922391 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:44.922260    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 03:23:44.926385 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:44.926362    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:45.011106 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:45.011073    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c6a6bf66-e3a4-46da-8a98-b60501b8fded-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-ff2jp\" (UID: \"c6a6bf66-e3a4-46da-8a98-b60501b8fded\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ff2jp"
Sep 09 03:23:45.011345 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:45.011326    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6a6bf66-e3a4-46da-8a98-b60501b8fded-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-ff2jp\" (UID: \"c6a6bf66-e3a4-46da-8a98-b60501b8fded\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ff2jp"
Sep 09 03:23:45.011476 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:45.011461    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb8st\" (UniqueName: \"kubernetes.io/projected/c6a6bf66-e3a4-46da-8a98-b60501b8fded-kube-api-access-pb8st\") pod \"kuadrant-console-plugin-fbdd4f445-ff2jp\" (UID: \"c6a6bf66-e3a4-46da-8a98-b60501b8fded\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ff2jp"
Sep 09 03:23:45.013211 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:45.013174    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-ff2jp"]
Sep 09 03:23:45.113032 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:45.112987    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c6a6bf66-e3a4-46da-8a98-b60501b8fded-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-ff2jp\" (UID: \"c6a6bf66-e3a4-46da-8a98-b60501b8fded\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ff2jp"
Sep 09 03:23:45.113225 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:45.113052    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6a6bf66-e3a4-46da-8a98-b60501b8fded-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-ff2jp\" (UID: \"c6a6bf66-e3a4-46da-8a98-b60501b8fded\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ff2jp"
Sep 09 03:23:45.113225 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:45.113091    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb8st\" (UniqueName: \"kubernetes.io/projected/c6a6bf66-e3a4-46da-8a98-b60501b8fded-kube-api-access-pb8st\") pod \"kuadrant-console-plugin-fbdd4f445-ff2jp\" (UID: \"c6a6bf66-e3a4-46da-8a98-b60501b8fded\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ff2jp"
Sep 09 03:23:45.114207 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:45.114175    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c6a6bf66-e3a4-46da-8a98-b60501b8fded-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-ff2jp\" (UID: \"c6a6bf66-e3a4-46da-8a98-b60501b8fded\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ff2jp"
Sep 09 03:23:45.123787 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:45.123756    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6a6bf66-e3a4-46da-8a98-b60501b8fded-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-ff2jp\" (UID: \"c6a6bf66-e3a4-46da-8a98-b60501b8fded\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ff2jp"
Sep 09 03:23:45.141790 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:45.141757    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb8st\" (UniqueName: \"kubernetes.io/projected/c6a6bf66-e3a4-46da-8a98-b60501b8fded-kube-api-access-pb8st\") pod \"kuadrant-console-plugin-fbdd4f445-ff2jp\" (UID: \"c6a6bf66-e3a4-46da-8a98-b60501b8fded\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ff2jp"
Sep 09 03:23:45.187248 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:45.187153    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:45.223977 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:45.223942    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ff2jp"
Sep 09 03:23:45.512430 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:45.512357    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-ff2jp"]
Sep 09 03:23:45.515503 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:23:45.515478    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6a6bf66_e3a4_46da_8a98_b60501b8fded.slice/crio-7813fe778e65ca165a7286ce1914386a16164fab8106e508dda2fdb90a9135dc WatchSource:0}: Error finding container 7813fe778e65ca165a7286ce1914386a16164fab8106e508dda2fdb90a9135dc: Status 404 returned error can't find the container with id 7813fe778e65ca165a7286ce1914386a16164fab8106e508dda2fdb90a9135dc
Sep 09 03:23:45.517274 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:45.517250    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:23:46.019643 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:46.019609    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ff2jp" event={"ID":"c6a6bf66-e3a4-46da-8a98-b60501b8fded","Type":"ContainerStarted","Data":"7813fe778e65ca165a7286ce1914386a16164fab8106e508dda2fdb90a9135dc"}
Sep 09 03:23:47.132681 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:47.132648    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:47.346815 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:47.346770    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:47.346995 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:47.346839    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:47.359469 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:47.359444    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:47.369045 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:47.369014    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:47.541938 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:47.541855    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:48.035191 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:48.035163    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69f9b5898b-92slb"
Sep 09 03:23:48.152062 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:48.152032    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c7bcd5696-r6zk4"]
Sep 09 03:23:48.270622 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:48.270549    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:48.459786 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:48.459755    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:48.679670 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:48.679632    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:48.857314 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:48.857276    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:51.053369 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:51.053332    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ff2jp" event={"ID":"c6a6bf66-e3a4-46da-8a98-b60501b8fded","Type":"ContainerStarted","Data":"1a1b391bb4523329d3a6b5e05da72ac23974dc27745fdd594c95f2a9677945b5"}
Sep 09 03:23:51.082858 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:51.082811    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ff2jp" podStartSLOduration=2.536351492 podStartE2EDuration="7.082796238s" totalImagesPullingTime="4.546444746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:23:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:23:51.080648451 +0000 UTC m=+648.217396316" watchObservedRunningTime="2026-09-09 03:23:51.082796238 +0000 UTC m=+648.219544103"
Sep 09 03:23:54.911917 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:54.911885    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:55.097635 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:55.097580    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:57.431561 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:57.431519    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:23:57.641986 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:23:57.641954    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:04.355513 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:04.355475    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:04.526283 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:04.526249    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:04.951863 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:04.951825    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:05.096447 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:05.096407    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:12.163812 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:12.163778    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:12.347089 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:12.347051    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:12.597204 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:12.597130    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:12.718445 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:12.718410    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:13.176692 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.176655    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c7bcd5696-r6zk4" podUID="9a78c4e9-bac0-4b29-be5d-a644ffe7261c" containerName="console" containerID="cri-o://1e68e38020d86ecb6fe0e450affb647614780b6af82f507d7f03e6e15dc03bda" gracePeriod=15
Sep 09 03:24:13.418731 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.418708    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:24:13.471345 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.471283    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-trusted-ca-bundle\") pod \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") "
Sep 09 03:24:13.471345 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.471329    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p97g7\" (UniqueName: \"kubernetes.io/projected/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-kube-api-access-p97g7\") pod \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") "
Sep 09 03:24:13.471506 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.471394    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-oauth-serving-cert\") pod \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") "
Sep 09 03:24:13.471506 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.471415    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-console-config\") pod \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") "
Sep 09 03:24:13.471506 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.471431    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-service-ca\") pod \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") "
Sep 09 03:24:13.471506 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.471476    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-console-serving-cert\") pod \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") "
Sep 09 03:24:13.471506 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.471503    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-console-oauth-config\") pod \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\" (UID: \"9a78c4e9-bac0-4b29-be5d-a644ffe7261c\") "
Sep 09 03:24:13.471782 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.471755    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9a78c4e9-bac0-4b29-be5d-a644ffe7261c" (UID: "9a78c4e9-bac0-4b29-be5d-a644ffe7261c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:24:13.471859 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.471821    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9a78c4e9-bac0-4b29-be5d-a644ffe7261c" (UID: "9a78c4e9-bac0-4b29-be5d-a644ffe7261c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:24:13.471905 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.471855    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-console-config" (OuterVolumeSpecName: "console-config") pod "9a78c4e9-bac0-4b29-be5d-a644ffe7261c" (UID: "9a78c4e9-bac0-4b29-be5d-a644ffe7261c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:24:13.471905 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.471846    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-trusted-ca-bundle\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:24:13.472169 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.472142    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-service-ca" (OuterVolumeSpecName: "service-ca") pod "9a78c4e9-bac0-4b29-be5d-a644ffe7261c" (UID: "9a78c4e9-bac0-4b29-be5d-a644ffe7261c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:24:13.473630 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.473607    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9a78c4e9-bac0-4b29-be5d-a644ffe7261c" (UID: "9a78c4e9-bac0-4b29-be5d-a644ffe7261c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:24:13.473735 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.473610    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-kube-api-access-p97g7" (OuterVolumeSpecName: "kube-api-access-p97g7") pod "9a78c4e9-bac0-4b29-be5d-a644ffe7261c" (UID: "9a78c4e9-bac0-4b29-be5d-a644ffe7261c"). InnerVolumeSpecName "kube-api-access-p97g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:24:13.473735 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.473706    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9a78c4e9-bac0-4b29-be5d-a644ffe7261c" (UID: "9a78c4e9-bac0-4b29-be5d-a644ffe7261c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:24:13.572811 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.572779    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-oauth-serving-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:24:13.572811 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.572806    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-console-config\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:24:13.572811 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.572817    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:24:13.573022 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.572825    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-console-serving-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:24:13.573022 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.572833    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-console-oauth-config\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:24:13.573022 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.572841    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p97g7\" (UniqueName: \"kubernetes.io/projected/9a78c4e9-bac0-4b29-be5d-a644ffe7261c-kube-api-access-p97g7\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:24:13.823777 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.823674    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:13.992772 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:13.992728    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:14.158282 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:14.158249    2560 generic.go:356] "Generic (PLEG): container finished" podID="9a78c4e9-bac0-4b29-be5d-a644ffe7261c" containerID="1e68e38020d86ecb6fe0e450affb647614780b6af82f507d7f03e6e15dc03bda" exitCode=0
Sep 09 03:24:14.158466 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:14.158298    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7bcd5696-r6zk4"
Sep 09 03:24:14.158466 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:14.158334    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7bcd5696-r6zk4" event={"ID":"9a78c4e9-bac0-4b29-be5d-a644ffe7261c","Type":"ContainerDied","Data":"1e68e38020d86ecb6fe0e450affb647614780b6af82f507d7f03e6e15dc03bda"}
Sep 09 03:24:14.158466 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:14.158392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7bcd5696-r6zk4" event={"ID":"9a78c4e9-bac0-4b29-be5d-a644ffe7261c","Type":"ContainerDied","Data":"d24f5cd78fd630c8cf6dc65b39140abea0cc0bdc25c4a7804bbe346e0272fb88"}
Sep 09 03:24:14.158466 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:14.158419    2560 scope.go:117] "RemoveContainer" containerID="1e68e38020d86ecb6fe0e450affb647614780b6af82f507d7f03e6e15dc03bda"
Sep 09 03:24:14.167003 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:14.166987    2560 scope.go:117] "RemoveContainer" containerID="1e68e38020d86ecb6fe0e450affb647614780b6af82f507d7f03e6e15dc03bda"
Sep 09 03:24:14.167276 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:24:14.167254    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e68e38020d86ecb6fe0e450affb647614780b6af82f507d7f03e6e15dc03bda\": container with ID starting with 1e68e38020d86ecb6fe0e450affb647614780b6af82f507d7f03e6e15dc03bda not found: ID does not exist" containerID="1e68e38020d86ecb6fe0e450affb647614780b6af82f507d7f03e6e15dc03bda"
Sep 09 03:24:14.167341 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:14.167283    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e68e38020d86ecb6fe0e450affb647614780b6af82f507d7f03e6e15dc03bda"} err="failed to get container status \"1e68e38020d86ecb6fe0e450affb647614780b6af82f507d7f03e6e15dc03bda\": rpc error: code = NotFound desc = could not find container \"1e68e38020d86ecb6fe0e450affb647614780b6af82f507d7f03e6e15dc03bda\": container with ID starting with 1e68e38020d86ecb6fe0e450affb647614780b6af82f507d7f03e6e15dc03bda not found: ID does not exist"
Sep 09 03:24:14.200541 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:14.200517    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c7bcd5696-r6zk4"]
Sep 09 03:24:14.204193 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:14.204170    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c7bcd5696-r6zk4"]
Sep 09 03:24:15.358862 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:15.358825    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a78c4e9-bac0-4b29-be5d-a644ffe7261c" path="/var/lib/kubelet/pods/9a78c4e9-bac0-4b29-be5d-a644ffe7261c/volumes"
Sep 09 03:24:20.648316 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:20.648279    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:20.784578 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:20.784523    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:23.696993 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:23.696951    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:23.828836 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:23.828798    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:29.010508 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:29.010473    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:29.196644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:29.196613    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:29.389165 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:29.389128    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:29.583502 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:29.583469    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:29.929191 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:29.929133    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:32.984008 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:32.983976    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:33.178087 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:33.178057    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:33.325039 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:33.324958    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:33.757942 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:33.757911    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:24:33.758255 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:33.758243    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a78c4e9-bac0-4b29-be5d-a644ffe7261c" containerName="console"
Sep 09 03:24:33.758300 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:33.758256    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a78c4e9-bac0-4b29-be5d-a644ffe7261c" containerName="console"
Sep 09 03:24:33.758335 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:33.758320    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a78c4e9-bac0-4b29-be5d-a644ffe7261c" containerName="console"
Sep 09 03:24:33.761194 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:33.761178    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"
Sep 09 03:24:33.771801 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:33.771782    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 03:24:33.778567 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:33.778544    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:24:33.816189 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:33.816160    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:24:33.849492 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:33.849466    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eed51b4a-fc6b-40a8-ae0c-d5ae12f45a34-config-file\") pod \"limitador-limitador-84994f7ffc-fdtxc\" (UID: \"eed51b4a-fc6b-40a8-ae0c-d5ae12f45a34\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"
Sep 09 03:24:33.849767 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:33.849501    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vxwm\" (UniqueName: \"kubernetes.io/projected/eed51b4a-fc6b-40a8-ae0c-d5ae12f45a34-kube-api-access-4vxwm\") pod \"limitador-limitador-84994f7ffc-fdtxc\" (UID: \"eed51b4a-fc6b-40a8-ae0c-d5ae12f45a34\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"
Sep 09 03:24:33.950927 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:33.950894    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eed51b4a-fc6b-40a8-ae0c-d5ae12f45a34-config-file\") pod \"limitador-limitador-84994f7ffc-fdtxc\" (UID: \"eed51b4a-fc6b-40a8-ae0c-d5ae12f45a34\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"
Sep 09 03:24:33.951079 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:33.950932    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vxwm\" (UniqueName: \"kubernetes.io/projected/eed51b4a-fc6b-40a8-ae0c-d5ae12f45a34-kube-api-access-4vxwm\") pod \"limitador-limitador-84994f7ffc-fdtxc\" (UID: \"eed51b4a-fc6b-40a8-ae0c-d5ae12f45a34\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"
Sep 09 03:24:33.951626 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:33.951590    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eed51b4a-fc6b-40a8-ae0c-d5ae12f45a34-config-file\") pod \"limitador-limitador-84994f7ffc-fdtxc\" (UID: \"eed51b4a-fc6b-40a8-ae0c-d5ae12f45a34\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"
Sep 09 03:24:33.962519 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:33.962491    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vxwm\" (UniqueName: \"kubernetes.io/projected/eed51b4a-fc6b-40a8-ae0c-d5ae12f45a34-kube-api-access-4vxwm\") pod \"limitador-limitador-84994f7ffc-fdtxc\" (UID: \"eed51b4a-fc6b-40a8-ae0c-d5ae12f45a34\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"
Sep 09 03:24:34.070785 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:34.070686    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"
Sep 09 03:24:34.280604 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:34.280571    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:34.378968 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:34.378942    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:24:34.380844 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:24:34.380817    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed51b4a_fc6b_40a8_ae0c_d5ae12f45a34.slice/crio-a3cedff28bfd302172b7afe4c7f02b13e6f8cd59dc95083fa1d076ed3902f9b4 WatchSource:0}: Error finding container a3cedff28bfd302172b7afe4c7f02b13e6f8cd59dc95083fa1d076ed3902f9b4: Status 404 returned error can't find the container with id a3cedff28bfd302172b7afe4c7f02b13e6f8cd59dc95083fa1d076ed3902f9b4
Sep 09 03:24:34.484702 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:34.484668    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:34.631279 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:34.631200    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-cqxdk"]
Sep 09 03:24:34.635766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:34.635748    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cqxdk"
Sep 09 03:24:34.639481 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:34.639464    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7dnvg\""
Sep 09 03:24:34.650435 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:34.650415    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cqxdk"]
Sep 09 03:24:34.759456 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:34.759422    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs6sg\" (UniqueName: \"kubernetes.io/projected/f26a6c71-0380-4b72-91d8-b3c3f124fd9b-kube-api-access-gs6sg\") pod \"authorino-59c974bd48-cqxdk\" (UID: \"f26a6c71-0380-4b72-91d8-b3c3f124fd9b\") " pod="kuadrant-system/authorino-59c974bd48-cqxdk"
Sep 09 03:24:34.861014 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:34.860917    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs6sg\" (UniqueName: \"kubernetes.io/projected/f26a6c71-0380-4b72-91d8-b3c3f124fd9b-kube-api-access-gs6sg\") pod \"authorino-59c974bd48-cqxdk\" (UID: \"f26a6c71-0380-4b72-91d8-b3c3f124fd9b\") " pod="kuadrant-system/authorino-59c974bd48-cqxdk"
Sep 09 03:24:34.873080 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:34.873054    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs6sg\" (UniqueName: \"kubernetes.io/projected/f26a6c71-0380-4b72-91d8-b3c3f124fd9b-kube-api-access-gs6sg\") pod \"authorino-59c974bd48-cqxdk\" (UID: \"f26a6c71-0380-4b72-91d8-b3c3f124fd9b\") " pod="kuadrant-system/authorino-59c974bd48-cqxdk"
Sep 09 03:24:34.945117 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:34.945039    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cqxdk"
Sep 09 03:24:35.235871 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:35.235824    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cqxdk"]
Sep 09 03:24:35.238019 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:24:35.237991    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf26a6c71_0380_4b72_91d8_b3c3f124fd9b.slice/crio-e261f9b06ef544cf99767a8d1ac4687d9c901e7b42199253003e31a9ef3ffc9a WatchSource:0}: Error finding container e261f9b06ef544cf99767a8d1ac4687d9c901e7b42199253003e31a9ef3ffc9a: Status 404 returned error can't find the container with id e261f9b06ef544cf99767a8d1ac4687d9c901e7b42199253003e31a9ef3ffc9a
Sep 09 03:24:35.253284 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:35.253258    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cqxdk" event={"ID":"f26a6c71-0380-4b72-91d8-b3c3f124fd9b","Type":"ContainerStarted","Data":"e261f9b06ef544cf99767a8d1ac4687d9c901e7b42199253003e31a9ef3ffc9a"}
Sep 09 03:24:35.254363 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:35.254341    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-fdtxc" event={"ID":"eed51b4a-fc6b-40a8-ae0c-d5ae12f45a34","Type":"ContainerStarted","Data":"a3cedff28bfd302172b7afe4c7f02b13e6f8cd59dc95083fa1d076ed3902f9b4"}
Sep 09 03:24:37.267649 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:37.267615    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-fdtxc" event={"ID":"eed51b4a-fc6b-40a8-ae0c-d5ae12f45a34","Type":"ContainerStarted","Data":"5f3f18558ddd6d9b0f2d0f1388be124dc4a66341498760e456a81133308d68ff"}
Sep 09 03:24:37.268121 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:37.267730    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"
Sep 09 03:24:37.314689 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:37.314633    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-fdtxc" podStartSLOduration=1.8866570870000001 podStartE2EDuration="4.314617858s" totalImagesPullingTime="2.427960771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:24:37.313527154 +0000 UTC m=+694.450275020" watchObservedRunningTime="2026-09-09 03:24:37.314617858 +0000 UTC m=+694.451365722"
Sep 09 03:24:38.274185 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:38.274079    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cqxdk" event={"ID":"f26a6c71-0380-4b72-91d8-b3c3f124fd9b","Type":"ContainerStarted","Data":"be625f5c28f4187f0c52da43a6ed3e2e07514b577854b22b020bb009a8024c28"}
Sep 09 03:24:48.275054 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:48.275016    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"
Sep 09 03:24:48.319268 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:48.319214    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-cqxdk" podStartSLOduration=11.639131584 podStartE2EDuration="14.319197529s" totalImagesPullingTime="2.680065945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:24:38.320063995 +0000 UTC m=+695.456811861" watchObservedRunningTime="2026-09-09 03:24:48.319197529 +0000 UTC m=+705.455945394"
Sep 09 03:24:54.257402 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:54.257369    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:54.411150 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:54.411119    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:54.595971 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:54.595888    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:54.843405 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:54.843372    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:56.936968 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:56.936890    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:24:57.041230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:24:57.041195    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:06.979422 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:06.979382    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:07.360835 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:07.360798    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:07.798612 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:07.798505    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:08.067544 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:08.067465    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7558766c89-ffbhr"]
Sep 09 03:25:08.075457 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:08.075434    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7558766c89-ffbhr"
Sep 09 03:25:08.078403 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:08.078386    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 03:25:08.086565 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:08.086541    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7558766c89-ffbhr"]
Sep 09 03:25:08.164502 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:08.164463    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa0a8cba-78f1-4376-b8cd-7d0d5aca737c-tls-cert\") pod \"authorino-7558766c89-ffbhr\" (UID: \"fa0a8cba-78f1-4376-b8cd-7d0d5aca737c\") " pod="kuadrant-system/authorino-7558766c89-ffbhr"
Sep 09 03:25:08.164502 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:08.164500    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvmxb\" (UniqueName: \"kubernetes.io/projected/fa0a8cba-78f1-4376-b8cd-7d0d5aca737c-kube-api-access-nvmxb\") pod \"authorino-7558766c89-ffbhr\" (UID: \"fa0a8cba-78f1-4376-b8cd-7d0d5aca737c\") " pod="kuadrant-system/authorino-7558766c89-ffbhr"
Sep 09 03:25:08.265361 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:08.265322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa0a8cba-78f1-4376-b8cd-7d0d5aca737c-tls-cert\") pod \"authorino-7558766c89-ffbhr\" (UID: \"fa0a8cba-78f1-4376-b8cd-7d0d5aca737c\") " pod="kuadrant-system/authorino-7558766c89-ffbhr"
Sep 09 03:25:08.265361 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:08.265360    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvmxb\" (UniqueName: \"kubernetes.io/projected/fa0a8cba-78f1-4376-b8cd-7d0d5aca737c-kube-api-access-nvmxb\") pod \"authorino-7558766c89-ffbhr\" (UID: \"fa0a8cba-78f1-4376-b8cd-7d0d5aca737c\") " pod="kuadrant-system/authorino-7558766c89-ffbhr"
Sep 09 03:25:08.267834 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:08.267811    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa0a8cba-78f1-4376-b8cd-7d0d5aca737c-tls-cert\") pod \"authorino-7558766c89-ffbhr\" (UID: \"fa0a8cba-78f1-4376-b8cd-7d0d5aca737c\") " pod="kuadrant-system/authorino-7558766c89-ffbhr"
Sep 09 03:25:08.276943 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:08.276922    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvmxb\" (UniqueName: \"kubernetes.io/projected/fa0a8cba-78f1-4376-b8cd-7d0d5aca737c-kube-api-access-nvmxb\") pod \"authorino-7558766c89-ffbhr\" (UID: \"fa0a8cba-78f1-4376-b8cd-7d0d5aca737c\") " pod="kuadrant-system/authorino-7558766c89-ffbhr"
Sep 09 03:25:08.384149 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:08.384116    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7558766c89-ffbhr"
Sep 09 03:25:08.671370 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:08.671336    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7558766c89-ffbhr"]
Sep 09 03:25:08.673828 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:25:08.673797    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa0a8cba_78f1_4376_b8cd_7d0d5aca737c.slice/crio-6079e54fa1cf90f220502b6fd2b7b0fb5a11f53f4a96f442783e6bac08981c07 WatchSource:0}: Error finding container 6079e54fa1cf90f220502b6fd2b7b0fb5a11f53f4a96f442783e6bac08981c07: Status 404 returned error can't find the container with id 6079e54fa1cf90f220502b6fd2b7b0fb5a11f53f4a96f442783e6bac08981c07
Sep 09 03:25:09.429382 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:09.429345    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7558766c89-ffbhr" event={"ID":"fa0a8cba-78f1-4376-b8cd-7d0d5aca737c","Type":"ContainerStarted","Data":"8ef557280bc9da569b2e1c92eb31a898f338d00f8e59e40b41620b22d2009e15"}
Sep 09 03:25:09.429382 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:09.429386    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7558766c89-ffbhr" event={"ID":"fa0a8cba-78f1-4376-b8cd-7d0d5aca737c","Type":"ContainerStarted","Data":"6079e54fa1cf90f220502b6fd2b7b0fb5a11f53f4a96f442783e6bac08981c07"}
Sep 09 03:25:09.448906 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:09.448849    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7558766c89-ffbhr" podStartSLOduration=0.872945594 podStartE2EDuration="1.448829537s" totalImagesPullingTime="575.883943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:25:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:25:09.447974093 +0000 UTC m=+726.584721958" watchObservedRunningTime="2026-09-09 03:25:09.448829537 +0000 UTC m=+726.585577403"
Sep 09 03:25:09.489186 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:09.489150    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cqxdk"]
Sep 09 03:25:09.489332 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:09.489316    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-cqxdk" podUID="f26a6c71-0380-4b72-91d8-b3c3f124fd9b" containerName="authorino" containerID="cri-o://be625f5c28f4187f0c52da43a6ed3e2e07514b577854b22b020bb009a8024c28" gracePeriod=30
Sep 09 03:25:09.732726 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:09.732702    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cqxdk"
Sep 09 03:25:09.879547 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:09.879515    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gs6sg\" (UniqueName: \"kubernetes.io/projected/f26a6c71-0380-4b72-91d8-b3c3f124fd9b-kube-api-access-gs6sg\") pod \"f26a6c71-0380-4b72-91d8-b3c3f124fd9b\" (UID: \"f26a6c71-0380-4b72-91d8-b3c3f124fd9b\") "
Sep 09 03:25:09.881674 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:09.881643    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26a6c71-0380-4b72-91d8-b3c3f124fd9b-kube-api-access-gs6sg" (OuterVolumeSpecName: "kube-api-access-gs6sg") pod "f26a6c71-0380-4b72-91d8-b3c3f124fd9b" (UID: "f26a6c71-0380-4b72-91d8-b3c3f124fd9b"). InnerVolumeSpecName "kube-api-access-gs6sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:25:09.980141 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:09.980117    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs6sg\" (UniqueName: \"kubernetes.io/projected/f26a6c71-0380-4b72-91d8-b3c3f124fd9b-kube-api-access-gs6sg\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:25:10.434983 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:10.434944    2560 generic.go:356] "Generic (PLEG): container finished" podID="f26a6c71-0380-4b72-91d8-b3c3f124fd9b" containerID="be625f5c28f4187f0c52da43a6ed3e2e07514b577854b22b020bb009a8024c28" exitCode=0
Sep 09 03:25:10.435374 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:10.434989    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cqxdk"
Sep 09 03:25:10.435374 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:10.435037    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cqxdk" event={"ID":"f26a6c71-0380-4b72-91d8-b3c3f124fd9b","Type":"ContainerDied","Data":"be625f5c28f4187f0c52da43a6ed3e2e07514b577854b22b020bb009a8024c28"}
Sep 09 03:25:10.435374 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:10.435086    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cqxdk" event={"ID":"f26a6c71-0380-4b72-91d8-b3c3f124fd9b","Type":"ContainerDied","Data":"e261f9b06ef544cf99767a8d1ac4687d9c901e7b42199253003e31a9ef3ffc9a"}
Sep 09 03:25:10.435374 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:10.435108    2560 scope.go:117] "RemoveContainer" containerID="be625f5c28f4187f0c52da43a6ed3e2e07514b577854b22b020bb009a8024c28"
Sep 09 03:25:10.443869 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:10.443853    2560 scope.go:117] "RemoveContainer" containerID="be625f5c28f4187f0c52da43a6ed3e2e07514b577854b22b020bb009a8024c28"
Sep 09 03:25:10.444123 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:25:10.444105    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be625f5c28f4187f0c52da43a6ed3e2e07514b577854b22b020bb009a8024c28\": container with ID starting with be625f5c28f4187f0c52da43a6ed3e2e07514b577854b22b020bb009a8024c28 not found: ID does not exist" containerID="be625f5c28f4187f0c52da43a6ed3e2e07514b577854b22b020bb009a8024c28"
Sep 09 03:25:10.444185 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:10.444138    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be625f5c28f4187f0c52da43a6ed3e2e07514b577854b22b020bb009a8024c28"} err="failed to get container status \"be625f5c28f4187f0c52da43a6ed3e2e07514b577854b22b020bb009a8024c28\": rpc error: code = NotFound desc = could not find container \"be625f5c28f4187f0c52da43a6ed3e2e07514b577854b22b020bb009a8024c28\": container with ID starting with be625f5c28f4187f0c52da43a6ed3e2e07514b577854b22b020bb009a8024c28 not found: ID does not exist"
Sep 09 03:25:10.463366 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:10.463337    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cqxdk"]
Sep 09 03:25:10.471870 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:10.471826    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cqxdk"]
Sep 09 03:25:11.358208 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:11.358166    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f26a6c71-0380-4b72-91d8-b3c3f124fd9b" path="/var/lib/kubelet/pods/f26a6c71-0380-4b72-91d8-b3c3f124fd9b/volumes"
Sep 09 03:25:15.138218 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:15.138184    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:15.307556 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:15.307525    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:15.461775 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:15.461689    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:17.849029 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:17.848995    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:18.151016 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:18.150977    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:48.767141 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:48.767107    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:48.943980 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:48.943942    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:50.167101 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:50.167065    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:50.416714 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:50.416683    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:50.631399 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:50.631367    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:50.807890 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:50.807854    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:51.318726 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:51.318696    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:51.725269 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:51.725229    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:25:51.916743 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:25:51.916708    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:26:01.175767 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:01.175730    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:01.330076 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:01.330041    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:18.462888 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:18.462855    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:18.737569 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:18.737475    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:19.378786 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:19.378748    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:26:31.011278 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:31.011199    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:31.740689 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:31.740654    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:26:39.856719 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:39.856681    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"]
Sep 09 03:26:39.857152 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:39.857037    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f26a6c71-0380-4b72-91d8-b3c3f124fd9b" containerName="authorino"
Sep 09 03:26:39.857152 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:39.857049    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f26a6c71-0380-4b72-91d8-b3c3f124fd9b" containerName="authorino"
Sep 09 03:26:39.857152 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:39.857105    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f26a6c71-0380-4b72-91d8-b3c3f124fd9b" containerName="authorino"
Sep 09 03:26:39.859982 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:39.859966    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:39.868363 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:39.868344    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 03:26:39.868648 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:39.868633    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9xhf2\""
Sep 09 03:26:39.869507 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:39.869491    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 09 03:26:39.871659 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:39.871640    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 03:26:39.917296 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:39.917262    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"]
Sep 09 03:26:39.975868 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:39.975833    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:39.993012 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:39.992989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa26adec-42ee-4838-b48c-4cdda3fc6a60-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6\" (UID: \"fa26adec-42ee-4838-b48c-4cdda3fc6a60\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:39.993160 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:39.993023    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa26adec-42ee-4838-b48c-4cdda3fc6a60-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6\" (UID: \"fa26adec-42ee-4838-b48c-4cdda3fc6a60\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:39.993160 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:39.993062    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa26adec-42ee-4838-b48c-4cdda3fc6a60-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6\" (UID: \"fa26adec-42ee-4838-b48c-4cdda3fc6a60\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:39.993160 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:39.993094    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa26adec-42ee-4838-b48c-4cdda3fc6a60-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6\" (UID: \"fa26adec-42ee-4838-b48c-4cdda3fc6a60\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:39.993160 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:39.993128    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa26adec-42ee-4838-b48c-4cdda3fc6a60-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6\" (UID: \"fa26adec-42ee-4838-b48c-4cdda3fc6a60\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:39.993307 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:39.993180    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8hpj\" (UniqueName: \"kubernetes.io/projected/fa26adec-42ee-4838-b48c-4cdda3fc6a60-kube-api-access-x8hpj\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6\" (UID: \"fa26adec-42ee-4838-b48c-4cdda3fc6a60\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:40.093909 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:40.093873    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa26adec-42ee-4838-b48c-4cdda3fc6a60-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6\" (UID: \"fa26adec-42ee-4838-b48c-4cdda3fc6a60\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:40.094073 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:40.093915    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa26adec-42ee-4838-b48c-4cdda3fc6a60-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6\" (UID: \"fa26adec-42ee-4838-b48c-4cdda3fc6a60\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:40.094073 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:40.093957    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa26adec-42ee-4838-b48c-4cdda3fc6a60-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6\" (UID: \"fa26adec-42ee-4838-b48c-4cdda3fc6a60\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:40.094073 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:40.093976    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa26adec-42ee-4838-b48c-4cdda3fc6a60-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6\" (UID: \"fa26adec-42ee-4838-b48c-4cdda3fc6a60\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:40.094073 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:40.093997    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa26adec-42ee-4838-b48c-4cdda3fc6a60-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6\" (UID: \"fa26adec-42ee-4838-b48c-4cdda3fc6a60\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:40.094073 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:40.094032    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8hpj\" (UniqueName: \"kubernetes.io/projected/fa26adec-42ee-4838-b48c-4cdda3fc6a60-kube-api-access-x8hpj\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6\" (UID: \"fa26adec-42ee-4838-b48c-4cdda3fc6a60\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:40.094381 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:40.094347    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa26adec-42ee-4838-b48c-4cdda3fc6a60-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6\" (UID: \"fa26adec-42ee-4838-b48c-4cdda3fc6a60\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:40.094381 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:40.094371    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa26adec-42ee-4838-b48c-4cdda3fc6a60-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6\" (UID: \"fa26adec-42ee-4838-b48c-4cdda3fc6a60\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:40.094483 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:40.094420    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa26adec-42ee-4838-b48c-4cdda3fc6a60-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6\" (UID: \"fa26adec-42ee-4838-b48c-4cdda3fc6a60\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:40.096220 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:40.096201    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa26adec-42ee-4838-b48c-4cdda3fc6a60-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6\" (UID: \"fa26adec-42ee-4838-b48c-4cdda3fc6a60\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:40.096583 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:40.096564    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa26adec-42ee-4838-b48c-4cdda3fc6a60-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6\" (UID: \"fa26adec-42ee-4838-b48c-4cdda3fc6a60\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:40.119200 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:40.119141    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8hpj\" (UniqueName: \"kubernetes.io/projected/fa26adec-42ee-4838-b48c-4cdda3fc6a60-kube-api-access-x8hpj\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6\" (UID: \"fa26adec-42ee-4838-b48c-4cdda3fc6a60\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:40.170166 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:40.170131    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:40.467564 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:40.467539    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"]
Sep 09 03:26:40.469426 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:26:40.469400    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa26adec_42ee_4838_b48c_4cdda3fc6a60.slice/crio-f4396992f4116262f38c3ce5040844a1af35e07bbf558f5c234322cae7d4e647 WatchSource:0}: Error finding container f4396992f4116262f38c3ce5040844a1af35e07bbf558f5c234322cae7d4e647: Status 404 returned error can't find the container with id f4396992f4116262f38c3ce5040844a1af35e07bbf558f5c234322cae7d4e647
Sep 09 03:26:40.883193 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:40.883156    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6" event={"ID":"fa26adec-42ee-4838-b48c-4cdda3fc6a60","Type":"ContainerStarted","Data":"f4396992f4116262f38c3ce5040844a1af35e07bbf558f5c234322cae7d4e647"}
Sep 09 03:26:40.953347 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:40.953227    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:26:41.889585 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:41.889550    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6" event={"ID":"fa26adec-42ee-4838-b48c-4cdda3fc6a60","Type":"ContainerStarted","Data":"903c19361e42f3a996b0335b9c563439cfcea33b3bcd92ea3a0e3c53cfa5b23b"}
Sep 09 03:26:41.927116 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:41.927063    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6" podStartSLOduration=1.868794441 podStartE2EDuration="2.927046658s" totalImagesPullingTime="1.058252217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:26:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:26:41.925541829 +0000 UTC m=+819.062289697" watchObservedRunningTime="2026-09-09 03:26:41.927046658 +0000 UTC m=+819.063794522"
Sep 09 03:26:48.493503 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:48.493470    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:49.831859 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:49.831822    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:49.960328 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:49.960292    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:26:49.980810 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:49.980774    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:50.171363 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:50.171264    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:50.171537 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:50.171406    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:50.184436 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:50.184405    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:50.943086 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:50.943061    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6"
Sep 09 03:26:53.659014 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.658972    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"]
Sep 09 03:26:53.662707 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.662685    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.665566 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.665546    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 09 03:26:53.691387 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.691364    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"]
Sep 09 03:26:53.800033 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.799999    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:53.823992 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.823961    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c0884da-95f2-4338-ae6b-ffbf5b9d7267-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq\" (UID: \"7c0884da-95f2-4338-ae6b-ffbf5b9d7267\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.824124 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.824032    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c0884da-95f2-4338-ae6b-ffbf5b9d7267-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq\" (UID: \"7c0884da-95f2-4338-ae6b-ffbf5b9d7267\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.824124 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.824086    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c0884da-95f2-4338-ae6b-ffbf5b9d7267-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq\" (UID: \"7c0884da-95f2-4338-ae6b-ffbf5b9d7267\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.824124 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.824108    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c0884da-95f2-4338-ae6b-ffbf5b9d7267-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq\" (UID: \"7c0884da-95f2-4338-ae6b-ffbf5b9d7267\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.824231 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.824130    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw9vq\" (UniqueName: \"kubernetes.io/projected/7c0884da-95f2-4338-ae6b-ffbf5b9d7267-kube-api-access-gw9vq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq\" (UID: \"7c0884da-95f2-4338-ae6b-ffbf5b9d7267\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.824266 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.824239    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c0884da-95f2-4338-ae6b-ffbf5b9d7267-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq\" (UID: \"7c0884da-95f2-4338-ae6b-ffbf5b9d7267\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.925051 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.924948    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw9vq\" (UniqueName: \"kubernetes.io/projected/7c0884da-95f2-4338-ae6b-ffbf5b9d7267-kube-api-access-gw9vq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq\" (UID: \"7c0884da-95f2-4338-ae6b-ffbf5b9d7267\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.925238 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.925060    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c0884da-95f2-4338-ae6b-ffbf5b9d7267-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq\" (UID: \"7c0884da-95f2-4338-ae6b-ffbf5b9d7267\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.925238 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.925099    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c0884da-95f2-4338-ae6b-ffbf5b9d7267-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq\" (UID: \"7c0884da-95f2-4338-ae6b-ffbf5b9d7267\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.925238 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.925139    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c0884da-95f2-4338-ae6b-ffbf5b9d7267-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq\" (UID: \"7c0884da-95f2-4338-ae6b-ffbf5b9d7267\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.925238 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.925171    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c0884da-95f2-4338-ae6b-ffbf5b9d7267-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq\" (UID: \"7c0884da-95f2-4338-ae6b-ffbf5b9d7267\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.925238 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.925195    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c0884da-95f2-4338-ae6b-ffbf5b9d7267-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq\" (UID: \"7c0884da-95f2-4338-ae6b-ffbf5b9d7267\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.925503 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.925438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c0884da-95f2-4338-ae6b-ffbf5b9d7267-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq\" (UID: \"7c0884da-95f2-4338-ae6b-ffbf5b9d7267\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.925503 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.925483    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c0884da-95f2-4338-ae6b-ffbf5b9d7267-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq\" (UID: \"7c0884da-95f2-4338-ae6b-ffbf5b9d7267\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.925640 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.925559    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c0884da-95f2-4338-ae6b-ffbf5b9d7267-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq\" (UID: \"7c0884da-95f2-4338-ae6b-ffbf5b9d7267\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.927631 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.927612    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c0884da-95f2-4338-ae6b-ffbf5b9d7267-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq\" (UID: \"7c0884da-95f2-4338-ae6b-ffbf5b9d7267\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.927733 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.927717    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c0884da-95f2-4338-ae6b-ffbf5b9d7267-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq\" (UID: \"7c0884da-95f2-4338-ae6b-ffbf5b9d7267\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.939905 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.939884    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw9vq\" (UniqueName: \"kubernetes.io/projected/7c0884da-95f2-4338-ae6b-ffbf5b9d7267-kube-api-access-gw9vq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq\" (UID: \"7c0884da-95f2-4338-ae6b-ffbf5b9d7267\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:53.973535 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:53.973512    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:26:54.311425 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:54.311373    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"]
Sep 09 03:26:54.313856 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:26:54.313825    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0884da_95f2_4338_ae6b_ffbf5b9d7267.slice/crio-f3b6c0b80b00098d77780ff3842e87513baa2165e2c4235c0a5241e39003c2ca WatchSource:0}: Error finding container f3b6c0b80b00098d77780ff3842e87513baa2165e2c4235c0a5241e39003c2ca: Status 404 returned error can't find the container with id f3b6c0b80b00098d77780ff3842e87513baa2165e2c4235c0a5241e39003c2ca
Sep 09 03:26:54.854935 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:54.854902    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:54.952433 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:54.952366    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq" event={"ID":"7c0884da-95f2-4338-ae6b-ffbf5b9d7267","Type":"ContainerStarted","Data":"582dd9eb3214f8d1f8e6bb0e9ca073a110975f7f0d97e4f4219706e3675c91c4"}
Sep 09 03:26:54.952588 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:54.952461    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq" event={"ID":"7c0884da-95f2-4338-ae6b-ffbf5b9d7267","Type":"ContainerStarted","Data":"f3b6c0b80b00098d77780ff3842e87513baa2165e2c4235c0a5241e39003c2ca"}
Sep 09 03:26:54.988651 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:54.988584    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq" podStartSLOduration=1.776990433 podStartE2EDuration="1.988567985s" totalImagesPullingTime="211.577552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:26:54.986116831 +0000 UTC m=+832.122864694" watchObservedRunningTime="2026-09-09 03:26:54.988567985 +0000 UTC m=+832.125315849"
Sep 09 03:26:55.029984 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:55.029953    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:26:55.699022 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:55.698987    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:26:59.722263 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:26:59.722210    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:27:02.101093 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:27:02.101057    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:27:03.974331 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:27:03.974293    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:27:03.974789 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:27:03.974347    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:27:03.986684 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:27:03.986654    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:27:04.009422 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:27:04.009392    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq"
Sep 09 03:27:24.892892 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:27:24.892855    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:27:27.267922 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:27:27.267883    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:28:09.404830 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:09.404802    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:28:09.432745 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:09.432722    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:28:52.143639 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:52.143606    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-857c4ff9cb-c76z4"]
Sep 09 03:28:52.147132 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:52.147111    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-857c4ff9cb-c76z4"
Sep 09 03:28:52.173900 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:52.173871    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-857c4ff9cb-c76z4"]
Sep 09 03:28:52.284051 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:52.284019    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36e730bb-c479-4b26-b925-30ac5137dcd3-tls-cert\") pod \"authorino-857c4ff9cb-c76z4\" (UID: \"36e730bb-c479-4b26-b925-30ac5137dcd3\") " pod="kuadrant-system/authorino-857c4ff9cb-c76z4"
Sep 09 03:28:52.284219 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:52.284079    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9c4c\" (UniqueName: \"kubernetes.io/projected/36e730bb-c479-4b26-b925-30ac5137dcd3-kube-api-access-r9c4c\") pod \"authorino-857c4ff9cb-c76z4\" (UID: \"36e730bb-c479-4b26-b925-30ac5137dcd3\") " pod="kuadrant-system/authorino-857c4ff9cb-c76z4"
Sep 09 03:28:52.384619 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:52.384571    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36e730bb-c479-4b26-b925-30ac5137dcd3-tls-cert\") pod \"authorino-857c4ff9cb-c76z4\" (UID: \"36e730bb-c479-4b26-b925-30ac5137dcd3\") " pod="kuadrant-system/authorino-857c4ff9cb-c76z4"
Sep 09 03:28:52.384830 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:52.384653    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9c4c\" (UniqueName: \"kubernetes.io/projected/36e730bb-c479-4b26-b925-30ac5137dcd3-kube-api-access-r9c4c\") pod \"authorino-857c4ff9cb-c76z4\" (UID: \"36e730bb-c479-4b26-b925-30ac5137dcd3\") " pod="kuadrant-system/authorino-857c4ff9cb-c76z4"
Sep 09 03:28:52.387213 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:52.387187    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36e730bb-c479-4b26-b925-30ac5137dcd3-tls-cert\") pod \"authorino-857c4ff9cb-c76z4\" (UID: \"36e730bb-c479-4b26-b925-30ac5137dcd3\") " pod="kuadrant-system/authorino-857c4ff9cb-c76z4"
Sep 09 03:28:52.395317 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:52.395236    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9c4c\" (UniqueName: \"kubernetes.io/projected/36e730bb-c479-4b26-b925-30ac5137dcd3-kube-api-access-r9c4c\") pod \"authorino-857c4ff9cb-c76z4\" (UID: \"36e730bb-c479-4b26-b925-30ac5137dcd3\") " pod="kuadrant-system/authorino-857c4ff9cb-c76z4"
Sep 09 03:28:52.456180 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:52.456143    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-857c4ff9cb-c76z4"
Sep 09 03:28:52.754052 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:52.753967    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-857c4ff9cb-c76z4"]
Sep 09 03:28:52.758073 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:28:52.758048    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36e730bb_c479_4b26_b925_30ac5137dcd3.slice/crio-4cc044da40d91665b9d66f5da61ae69d00ec89cbd0cd35087f83313a3aaab0e1 WatchSource:0}: Error finding container 4cc044da40d91665b9d66f5da61ae69d00ec89cbd0cd35087f83313a3aaab0e1: Status 404 returned error can't find the container with id 4cc044da40d91665b9d66f5da61ae69d00ec89cbd0cd35087f83313a3aaab0e1
Sep 09 03:28:52.759771 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:52.759751    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:28:53.515822 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:53.515785    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-857c4ff9cb-c76z4" event={"ID":"36e730bb-c479-4b26-b925-30ac5137dcd3","Type":"ContainerStarted","Data":"4cc044da40d91665b9d66f5da61ae69d00ec89cbd0cd35087f83313a3aaab0e1"}
Sep 09 03:28:54.524150 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:54.524116    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-857c4ff9cb-c76z4" event={"ID":"36e730bb-c479-4b26-b925-30ac5137dcd3","Type":"ContainerStarted","Data":"fb8ca5cf96dbf8b4c4a210070b659f5d1a612c2e8c6a7ba5d68af5b3a36aef3f"}
Sep 09 03:28:54.567450 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:54.567394    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-857c4ff9cb-c76z4" podStartSLOduration=0.898757879 podStartE2EDuration="2.567377839s" totalImagesPullingTime="1.66861996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:28:54.563247348 +0000 UTC m=+951.699995212" watchObservedRunningTime="2026-09-09 03:28:54.567377839 +0000 UTC m=+951.704125704"
Sep 09 03:28:54.668159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:54.668128    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7558766c89-ffbhr"]
Sep 09 03:28:54.668328 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:54.668309    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7558766c89-ffbhr" podUID="fa0a8cba-78f1-4376-b8cd-7d0d5aca737c" containerName="authorino" containerID="cri-o://8ef557280bc9da569b2e1c92eb31a898f338d00f8e59e40b41620b22d2009e15" gracePeriod=30
Sep 09 03:28:54.915441 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:54.915415    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7558766c89-ffbhr"
Sep 09 03:28:55.010531 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:55.010433    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa0a8cba-78f1-4376-b8cd-7d0d5aca737c-tls-cert\") pod \"fa0a8cba-78f1-4376-b8cd-7d0d5aca737c\" (UID: \"fa0a8cba-78f1-4376-b8cd-7d0d5aca737c\") "
Sep 09 03:28:55.010724 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:55.010612    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvmxb\" (UniqueName: \"kubernetes.io/projected/fa0a8cba-78f1-4376-b8cd-7d0d5aca737c-kube-api-access-nvmxb\") pod \"fa0a8cba-78f1-4376-b8cd-7d0d5aca737c\" (UID: \"fa0a8cba-78f1-4376-b8cd-7d0d5aca737c\") "
Sep 09 03:28:55.012637 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:55.012608    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0a8cba-78f1-4376-b8cd-7d0d5aca737c-kube-api-access-nvmxb" (OuterVolumeSpecName: "kube-api-access-nvmxb") pod "fa0a8cba-78f1-4376-b8cd-7d0d5aca737c" (UID: "fa0a8cba-78f1-4376-b8cd-7d0d5aca737c"). InnerVolumeSpecName "kube-api-access-nvmxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:28:55.026703 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:55.026667    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0a8cba-78f1-4376-b8cd-7d0d5aca737c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "fa0a8cba-78f1-4376-b8cd-7d0d5aca737c" (UID: "fa0a8cba-78f1-4376-b8cd-7d0d5aca737c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:28:55.112619 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:55.112561    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvmxb\" (UniqueName: \"kubernetes.io/projected/fa0a8cba-78f1-4376-b8cd-7d0d5aca737c-kube-api-access-nvmxb\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:28:55.112778 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:55.112634    2560 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa0a8cba-78f1-4376-b8cd-7d0d5aca737c-tls-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:28:55.534111 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:55.534064    2560 generic.go:356] "Generic (PLEG): container finished" podID="fa0a8cba-78f1-4376-b8cd-7d0d5aca737c" containerID="8ef557280bc9da569b2e1c92eb31a898f338d00f8e59e40b41620b22d2009e15" exitCode=0
Sep 09 03:28:55.534610 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:55.534472    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7558766c89-ffbhr"
Sep 09 03:28:55.534787 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:55.534762    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7558766c89-ffbhr" event={"ID":"fa0a8cba-78f1-4376-b8cd-7d0d5aca737c","Type":"ContainerDied","Data":"8ef557280bc9da569b2e1c92eb31a898f338d00f8e59e40b41620b22d2009e15"}
Sep 09 03:28:55.534870 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:55.534802    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7558766c89-ffbhr" event={"ID":"fa0a8cba-78f1-4376-b8cd-7d0d5aca737c","Type":"ContainerDied","Data":"6079e54fa1cf90f220502b6fd2b7b0fb5a11f53f4a96f442783e6bac08981c07"}
Sep 09 03:28:55.534870 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:55.534824    2560 scope.go:117] "RemoveContainer" containerID="8ef557280bc9da569b2e1c92eb31a898f338d00f8e59e40b41620b22d2009e15"
Sep 09 03:28:55.545902 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:55.545881    2560 scope.go:117] "RemoveContainer" containerID="8ef557280bc9da569b2e1c92eb31a898f338d00f8e59e40b41620b22d2009e15"
Sep 09 03:28:55.546248 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:28:55.546224    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef557280bc9da569b2e1c92eb31a898f338d00f8e59e40b41620b22d2009e15\": container with ID starting with 8ef557280bc9da569b2e1c92eb31a898f338d00f8e59e40b41620b22d2009e15 not found: ID does not exist" containerID="8ef557280bc9da569b2e1c92eb31a898f338d00f8e59e40b41620b22d2009e15"
Sep 09 03:28:55.546322 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:55.546260    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef557280bc9da569b2e1c92eb31a898f338d00f8e59e40b41620b22d2009e15"} err="failed to get container status \"8ef557280bc9da569b2e1c92eb31a898f338d00f8e59e40b41620b22d2009e15\": rpc error: code = NotFound desc = could not find container \"8ef557280bc9da569b2e1c92eb31a898f338d00f8e59e40b41620b22d2009e15\": container with ID starting with 8ef557280bc9da569b2e1c92eb31a898f338d00f8e59e40b41620b22d2009e15 not found: ID does not exist"
Sep 09 03:28:55.611193 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:55.611158    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7558766c89-ffbhr"]
Sep 09 03:28:55.638456 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:55.638369    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7558766c89-ffbhr"]
Sep 09 03:28:55.805116 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:55.805076    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:28:57.358982 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:57.358946    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa0a8cba-78f1-4376-b8cd-7d0d5aca737c" path="/var/lib/kubelet/pods/fa0a8cba-78f1-4376-b8cd-7d0d5aca737c/volumes"
Sep 09 03:28:58.021151 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:28:58.021113    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:15.877091 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:15.877047    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:16.245283 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:16.245204    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:16.428839 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:16.428796    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:16.883557 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:16.883519    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:17.608235 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:17.608204    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:17.733495 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:17.733459    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:18.138214 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:18.138176    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7659-140687837591088"]
Sep 09 03:29:18.138791 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:18.138773    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0a8cba-78f1-4376-b8cd-7d0d5aca737c" containerName="authorino"
Sep 09 03:29:18.138881 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:18.138795    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0a8cba-78f1-4376-b8cd-7d0d5aca737c" containerName="authorino"
Sep 09 03:29:18.138936 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:18.138905    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0a8cba-78f1-4376-b8cd-7d0d5aca737c" containerName="authorino"
Sep 09 03:29:18.143169 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:18.143149    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-140687837591088"
Sep 09 03:29:18.146220 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:18.146203    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-k4rsk\""
Sep 09 03:29:18.152958 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:18.152938    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7659-140687837591088"]
Sep 09 03:29:18.182818 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:18.182780    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:18.223429 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:18.223394    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxctm\" (UniqueName: \"kubernetes.io/projected/3120e423-e46c-43f5-a529-4de1169d8888-kube-api-access-kxctm\") pod \"test-curl-7659-140687837591088\" (UID: \"3120e423-e46c-43f5-a529-4de1169d8888\") " pod="opendatahub/test-curl-7659-140687837591088"
Sep 09 03:29:18.324238 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:18.324204    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxctm\" (UniqueName: \"kubernetes.io/projected/3120e423-e46c-43f5-a529-4de1169d8888-kube-api-access-kxctm\") pod \"test-curl-7659-140687837591088\" (UID: \"3120e423-e46c-43f5-a529-4de1169d8888\") " pod="opendatahub/test-curl-7659-140687837591088"
Sep 09 03:29:18.336401 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:18.336375    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxctm\" (UniqueName: \"kubernetes.io/projected/3120e423-e46c-43f5-a529-4de1169d8888-kube-api-access-kxctm\") pod \"test-curl-7659-140687837591088\" (UID: \"3120e423-e46c-43f5-a529-4de1169d8888\") " pod="opendatahub/test-curl-7659-140687837591088"
Sep 09 03:29:18.452929 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:18.452851    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-140687837591088"
Sep 09 03:29:18.723563 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:18.723537    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7659-140687837591088"]
Sep 09 03:29:18.725143 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:29:18.725113    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3120e423_e46c_43f5_a529_4de1169d8888.slice/crio-f14ddc60f18984cefe0f9a59eec10e9ab6c1a4ad717532e6d85dbf4ee75f8772 WatchSource:0}: Error finding container f14ddc60f18984cefe0f9a59eec10e9ab6c1a4ad717532e6d85dbf4ee75f8772: Status 404 returned error can't find the container with id f14ddc60f18984cefe0f9a59eec10e9ab6c1a4ad717532e6d85dbf4ee75f8772
Sep 09 03:29:19.539688 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:19.539644    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:19.656742 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:19.656705    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7659-140687837591088" event={"ID":"3120e423-e46c-43f5-a529-4de1169d8888","Type":"ContainerStarted","Data":"f14ddc60f18984cefe0f9a59eec10e9ab6c1a4ad717532e6d85dbf4ee75f8772"}
Sep 09 03:29:19.731288 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:19.731245    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:29:19.830764 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:19.830669    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:20.662993 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:20.662957    2560 generic.go:356] "Generic (PLEG): container finished" podID="3120e423-e46c-43f5-a529-4de1169d8888" containerID="9dc8e7de7c498a853946a882b3fa5c9129d3120b2ab2858923613c19e4712749" exitCode=0
Sep 09 03:29:20.663360 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:20.663026    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7659-140687837591088" event={"ID":"3120e423-e46c-43f5-a529-4de1169d8888","Type":"ContainerDied","Data":"9dc8e7de7c498a853946a882b3fa5c9129d3120b2ab2858923613c19e4712749"}
Sep 09 03:29:20.870777 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:20.870742    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:20.988695 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:20.988608    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:21.216182 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:21.216148    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:21.533569 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:21.533535    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:21.639015 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:21.638982    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:21.799780 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:21.799719    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-140687837591088"
Sep 09 03:29:21.838811 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:21.838785    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7659-140687837591088_3120e423-e46c-43f5-a529-4de1169d8888/test-curl-7659-140687837591088/0.log"
Sep 09 03:29:21.858746 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:21.858718    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxctm\" (UniqueName: \"kubernetes.io/projected/3120e423-e46c-43f5-a529-4de1169d8888-kube-api-access-kxctm\") pod \"3120e423-e46c-43f5-a529-4de1169d8888\" (UID: \"3120e423-e46c-43f5-a529-4de1169d8888\") "
Sep 09 03:29:21.860902 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:21.860872    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3120e423-e46c-43f5-a529-4de1169d8888-kube-api-access-kxctm" (OuterVolumeSpecName: "kube-api-access-kxctm") pod "3120e423-e46c-43f5-a529-4de1169d8888" (UID: "3120e423-e46c-43f5-a529-4de1169d8888"). InnerVolumeSpecName "kube-api-access-kxctm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:29:21.874256 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:21.874227    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7659-140687837591088"]
Sep 09 03:29:21.884950 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:21.884921    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7659-140687837591088"]
Sep 09 03:29:21.960028 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:21.959995    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxctm\" (UniqueName: \"kubernetes.io/projected/3120e423-e46c-43f5-a529-4de1169d8888-kube-api-access-kxctm\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:29:22.025370 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:22.025335    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7659-140687837591728"]
Sep 09 03:29:22.025735 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:22.025719    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3120e423-e46c-43f5-a529-4de1169d8888" containerName="test-curl-7659-140687837591088"
Sep 09 03:29:22.025782 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:22.025737    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3120e423-e46c-43f5-a529-4de1169d8888" containerName="test-curl-7659-140687837591088"
Sep 09 03:29:22.025820 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:22.025811    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3120e423-e46c-43f5-a529-4de1169d8888" containerName="test-curl-7659-140687837591088"
Sep 09 03:29:22.028962 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:22.028947    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-140687837591728"
Sep 09 03:29:22.045431 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:22.045407    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7659-140687837591728"]
Sep 09 03:29:22.060827 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:22.060765    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbbgw\" (UniqueName: \"kubernetes.io/projected/eec76463-1b98-45a8-abb3-54f0cdbcdd3b-kube-api-access-dbbgw\") pod \"test-curl-7659-140687837591728\" (UID: \"eec76463-1b98-45a8-abb3-54f0cdbcdd3b\") " pod="opendatahub/test-curl-7659-140687837591728"
Sep 09 03:29:22.161676 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:22.161643    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbbgw\" (UniqueName: \"kubernetes.io/projected/eec76463-1b98-45a8-abb3-54f0cdbcdd3b-kube-api-access-dbbgw\") pod \"test-curl-7659-140687837591728\" (UID: \"eec76463-1b98-45a8-abb3-54f0cdbcdd3b\") " pod="opendatahub/test-curl-7659-140687837591728"
Sep 09 03:29:22.172946 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:22.172914    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbbgw\" (UniqueName: \"kubernetes.io/projected/eec76463-1b98-45a8-abb3-54f0cdbcdd3b-kube-api-access-dbbgw\") pod \"test-curl-7659-140687837591728\" (UID: \"eec76463-1b98-45a8-abb3-54f0cdbcdd3b\") " pod="opendatahub/test-curl-7659-140687837591728"
Sep 09 03:29:22.338179 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:22.338088    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-140687837591728"
Sep 09 03:29:22.607311 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:22.607250    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7659-140687837591728"]
Sep 09 03:29:22.609547 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:29:22.609522    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeec76463_1b98_45a8_abb3_54f0cdbcdd3b.slice/crio-50e213cbdbad4572bedad272667e729379ed1f938bc0293dba24d5b576156c06 WatchSource:0}: Error finding container 50e213cbdbad4572bedad272667e729379ed1f938bc0293dba24d5b576156c06: Status 404 returned error can't find the container with id 50e213cbdbad4572bedad272667e729379ed1f938bc0293dba24d5b576156c06
Sep 09 03:29:22.677114 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:22.677086    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-140687837591088"
Sep 09 03:29:22.677114 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:22.677099    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f14ddc60f18984cefe0f9a59eec10e9ab6c1a4ad717532e6d85dbf4ee75f8772"
Sep 09 03:29:22.678293 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:22.678268    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7659-140687837591728" event={"ID":"eec76463-1b98-45a8-abb3-54f0cdbcdd3b","Type":"ContainerStarted","Data":"50e213cbdbad4572bedad272667e729379ed1f938bc0293dba24d5b576156c06"}
Sep 09 03:29:22.687562 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:29:22.687535    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7659-140687837591088\" is forbidden: User \"system:node:ip-10-0-133-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-131.ec2.internal' and this object" podUID="3120e423-e46c-43f5-a529-4de1169d8888" pod="opendatahub/test-curl-7659-140687837591088"
Sep 09 03:29:23.357162 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:29:23.357122    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7659-140687837591088\" is forbidden: User \"system:node:ip-10-0-133-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-131.ec2.internal' and this object" podUID="3120e423-e46c-43f5-a529-4de1169d8888" pod="opendatahub/test-curl-7659-140687837591088"
Sep 09 03:29:23.358422 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:23.358399    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3120e423-e46c-43f5-a529-4de1169d8888" path="/var/lib/kubelet/pods/3120e423-e46c-43f5-a529-4de1169d8888/volumes"
Sep 09 03:29:23.685284 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:23.685246    2560 generic.go:356] "Generic (PLEG): container finished" podID="eec76463-1b98-45a8-abb3-54f0cdbcdd3b" containerID="4f50b8a9613b559e71d82c9766d1fa1a204078ab96f8e246744a2d66c82bf3e5" exitCode=0
Sep 09 03:29:23.685436 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:23.685329    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7659-140687837591728" event={"ID":"eec76463-1b98-45a8-abb3-54f0cdbcdd3b","Type":"ContainerDied","Data":"4f50b8a9613b559e71d82c9766d1fa1a204078ab96f8e246744a2d66c82bf3e5"}
Sep 09 03:29:24.654657 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:24.654624    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:24.763409 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:24.763376    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:24.822072 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:24.822046    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-140687837591728"
Sep 09 03:29:24.851293 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:24.851264    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7659-140687837591728_eec76463-1b98-45a8-abb3-54f0cdbcdd3b/test-curl-7659-140687837591728/0.log"
Sep 09 03:29:24.881913 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:24.881875    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7659-140687837591728"]
Sep 09 03:29:24.886160 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:24.886128    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbbgw\" (UniqueName: \"kubernetes.io/projected/eec76463-1b98-45a8-abb3-54f0cdbcdd3b-kube-api-access-dbbgw\") pod \"eec76463-1b98-45a8-abb3-54f0cdbcdd3b\" (UID: \"eec76463-1b98-45a8-abb3-54f0cdbcdd3b\") "
Sep 09 03:29:24.888363 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:24.888332    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec76463-1b98-45a8-abb3-54f0cdbcdd3b-kube-api-access-dbbgw" (OuterVolumeSpecName: "kube-api-access-dbbgw") pod "eec76463-1b98-45a8-abb3-54f0cdbcdd3b" (UID: "eec76463-1b98-45a8-abb3-54f0cdbcdd3b"). InnerVolumeSpecName "kube-api-access-dbbgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:29:24.889508 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:24.889489    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7659-140687837591728"]
Sep 09 03:29:24.916159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:24.916065    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:24.987188 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:24.987154    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbbgw\" (UniqueName: \"kubernetes.io/projected/eec76463-1b98-45a8-abb3-54f0cdbcdd3b-kube-api-access-dbbgw\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:29:25.278546 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.278469    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"]
Sep 09 03:29:25.279065 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.279043    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eec76463-1b98-45a8-abb3-54f0cdbcdd3b" containerName="test-curl-7659-140687837591728"
Sep 09 03:29:25.279116 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.279070    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eec76463-1b98-45a8-abb3-54f0cdbcdd3b" containerName="test-curl-7659-140687837591728"
Sep 09 03:29:25.279176 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.279164    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eec76463-1b98-45a8-abb3-54f0cdbcdd3b" containerName="test-curl-7659-140687837591728"
Sep 09 03:29:25.282298 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.282275    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.289025 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.288966    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 03:29:25.290277 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.290234    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9c982818-openshift-default-dockercfg-g7l2m\""
Sep 09 03:29:25.316394 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.316365    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"]
Sep 09 03:29:25.360338 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.360302    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eec76463-1b98-45a8-abb3-54f0cdbcdd3b" path="/var/lib/kubelet/pods/eec76463-1b98-45a8-abb3-54f0cdbcdd3b/volumes"
Sep 09 03:29:25.369262 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.369238    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:25.389972 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.389925    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-podinfo\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.389972 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.389967    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-token\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.390216 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.390019    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-data\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.390216 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.390086    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-envoy\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.390216 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.390154    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-workload-socket\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.390394 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.390218    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-credential-socket\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.390394 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.390247    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-workload-certs\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.390394 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.390285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istiod-ca-cert\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.390394 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.390312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v724\" (UniqueName: \"kubernetes.io/projected/9c65e186-03b1-4ad0-80d6-e2eacf40e324-kube-api-access-5v724\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.491823 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.491782    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-podinfo\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.491823 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.491819    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-token\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.492071 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.491946    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-data\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.492071 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.491985    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-envoy\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.492071 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.492016    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-workload-socket\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.492071 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.492049    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-credential-socket\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.492071 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.492063    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-workload-certs\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.492325 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.492086    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istiod-ca-cert\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.492325 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.492103    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v724\" (UniqueName: \"kubernetes.io/projected/9c65e186-03b1-4ad0-80d6-e2eacf40e324-kube-api-access-5v724\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.492511 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.492488    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-workload-certs\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.492577 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.492543    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-data\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.492663 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.492573    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-workload-socket\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.492736 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.492718    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-credential-socket\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.492854 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.492834    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istiod-ca-cert\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.494296 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.494275    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-envoy\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.494541 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.494524    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-podinfo\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.504790 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.504764    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:25.514176 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.514146    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-token\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.514289 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.514230    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v724\" (UniqueName: \"kubernetes.io/projected/9c65e186-03b1-4ad0-80d6-e2eacf40e324-kube-api-access-5v724\") pod \"e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") " pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.593228 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.593130    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:25.619492 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.619460    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:25.696754 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.696728    2560 scope.go:117] "RemoveContainer" containerID="4f50b8a9613b559e71d82c9766d1fa1a204078ab96f8e246744a2d66c82bf3e5"
Sep 09 03:29:25.696754 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.696742    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-140687837591728"
Sep 09 03:29:25.876587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.876543    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:25.891936 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:25.891898    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"]
Sep 09 03:29:25.897936 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:29:25.897899    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c65e186_03b1_4ad0_80d6_e2eacf40e324.slice/crio-f79b9ecf6cd722626d05728dab9d54b7f44aaeed60c945d26e9c576e1388f9fb WatchSource:0}: Error finding container f79b9ecf6cd722626d05728dab9d54b7f44aaeed60c945d26e9c576e1388f9fb: Status 404 returned error can't find the container with id f79b9ecf6cd722626d05728dab9d54b7f44aaeed60c945d26e9c576e1388f9fb
Sep 09 03:29:26.414308 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:26.414273    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:26.713054 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:26.712965    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq" event={"ID":"9c65e186-03b1-4ad0-80d6-e2eacf40e324","Type":"ContainerStarted","Data":"f79b9ecf6cd722626d05728dab9d54b7f44aaeed60c945d26e9c576e1388f9fb"}
Sep 09 03:29:28.492476 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:28.492439    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:29:28.492778 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:28.492513    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:29:28.492778 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:28.492542    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:29:28.726381 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:28.726336    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq" event={"ID":"9c65e186-03b1-4ad0-80d6-e2eacf40e324","Type":"ContainerStarted","Data":"6ee71488fcb02ecdc4f17aeb8a5209d2486d3ba72fdf7886c2a737bd974236ea"}
Sep 09 03:29:28.763209 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:28.762925    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq" podStartSLOduration=1.171131044 podStartE2EDuration="3.762904646s" totalImagesPullingTime="2.591773602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:29:28.76177335 +0000 UTC m=+985.898521217" watchObservedRunningTime="2026-09-09 03:29:28.762904646 +0000 UTC m=+985.899652512"
Sep 09 03:29:29.496948 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:29.496914    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:29.593701 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:29.593651    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:29.598912 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:29.598886    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:29.630478 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:29.630445    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:29.731879 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:29.731848    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:29.732760 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:29.732744    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:30.211066 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:30.211035    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:30.362942 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:30.362915    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:30.949075 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:30.949035    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:31.047666 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:31.047630    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:31.317782 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:31.317682    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:31.597477 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:31.597386    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:31.712225 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:31.712192    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:32.091634 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.091588    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"]
Sep 09 03:29:32.095393 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.095371    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:29:32.098664 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.098496    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 03:29:32.098664 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.098501    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9c982818-serving-cert\""
Sep 09 03:29:32.098838 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.098701    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-phvzt\""
Sep 09 03:29:32.098927 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.098906    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 03:29:32.099067 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.099023    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 03:29:32.111397 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.111338    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"]
Sep 09 03:29:32.147621 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.147562    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:32.157791 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.157756    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8082450-ffff-407a-8b28-0efcc925727f-openshift-service-ca\") pod \"maas-api-e2e-ait-9c982818-6657f4cf57-662kw\" (UID: \"f8082450-ffff-407a-8b28-0efcc925727f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:29:32.157925 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.157829    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5mh\" (UniqueName: \"kubernetes.io/projected/f8082450-ffff-407a-8b28-0efcc925727f-kube-api-access-gg5mh\") pod \"maas-api-e2e-ait-9c982818-6657f4cf57-662kw\" (UID: \"f8082450-ffff-407a-8b28-0efcc925727f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:29:32.157925 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.157865    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8082450-ffff-407a-8b28-0efcc925727f-maas-api-tls\") pod \"maas-api-e2e-ait-9c982818-6657f4cf57-662kw\" (UID: \"f8082450-ffff-407a-8b28-0efcc925727f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:29:32.157925 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.157882    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8082450-ffff-407a-8b28-0efcc925727f-metrics-certs\") pod \"maas-api-e2e-ait-9c982818-6657f4cf57-662kw\" (UID: \"f8082450-ffff-407a-8b28-0efcc925727f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:29:32.259373 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.259333    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8082450-ffff-407a-8b28-0efcc925727f-openshift-service-ca\") pod \"maas-api-e2e-ait-9c982818-6657f4cf57-662kw\" (UID: \"f8082450-ffff-407a-8b28-0efcc925727f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:29:32.259554 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.259385    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5mh\" (UniqueName: \"kubernetes.io/projected/f8082450-ffff-407a-8b28-0efcc925727f-kube-api-access-gg5mh\") pod \"maas-api-e2e-ait-9c982818-6657f4cf57-662kw\" (UID: \"f8082450-ffff-407a-8b28-0efcc925727f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:29:32.259554 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.259426    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8082450-ffff-407a-8b28-0efcc925727f-maas-api-tls\") pod \"maas-api-e2e-ait-9c982818-6657f4cf57-662kw\" (UID: \"f8082450-ffff-407a-8b28-0efcc925727f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:29:32.259554 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.259456    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8082450-ffff-407a-8b28-0efcc925727f-metrics-certs\") pod \"maas-api-e2e-ait-9c982818-6657f4cf57-662kw\" (UID: \"f8082450-ffff-407a-8b28-0efcc925727f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:29:32.259771 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:29:32.259557    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-serving-cert: secret "maas-api-e2e-ait-9c982818-serving-cert" not found
Sep 09 03:29:32.259771 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:29:32.259673    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8082450-ffff-407a-8b28-0efcc925727f-maas-api-tls podName:f8082450-ffff-407a-8b28-0efcc925727f nodeName:}" failed. No retries permitted until 2026-09-09 03:29:32.759655412 +0000 UTC m=+989.896403258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8082450-ffff-407a-8b28-0efcc925727f-maas-api-tls") pod "maas-api-e2e-ait-9c982818-6657f4cf57-662kw" (UID: "f8082450-ffff-407a-8b28-0efcc925727f") : secret "maas-api-e2e-ait-9c982818-serving-cert" not found
Sep 09 03:29:32.260118 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.260092    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8082450-ffff-407a-8b28-0efcc925727f-openshift-service-ca\") pod \"maas-api-e2e-ait-9c982818-6657f4cf57-662kw\" (UID: \"f8082450-ffff-407a-8b28-0efcc925727f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:29:32.261299 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.261271    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:32.262431 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.262411    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8082450-ffff-407a-8b28-0efcc925727f-metrics-certs\") pod \"maas-api-e2e-ait-9c982818-6657f4cf57-662kw\" (UID: \"f8082450-ffff-407a-8b28-0efcc925727f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:29:32.270337 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.270315    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5mh\" (UniqueName: \"kubernetes.io/projected/f8082450-ffff-407a-8b28-0efcc925727f-kube-api-access-gg5mh\") pod \"maas-api-e2e-ait-9c982818-6657f4cf57-662kw\" (UID: \"f8082450-ffff-407a-8b28-0efcc925727f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:29:32.488920 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.488884    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:32.765230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.765142    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8082450-ffff-407a-8b28-0efcc925727f-maas-api-tls\") pod \"maas-api-e2e-ait-9c982818-6657f4cf57-662kw\" (UID: \"f8082450-ffff-407a-8b28-0efcc925727f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:29:32.767758 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.767729    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8082450-ffff-407a-8b28-0efcc925727f-maas-api-tls\") pod \"maas-api-e2e-ait-9c982818-6657f4cf57-662kw\" (UID: \"f8082450-ffff-407a-8b28-0efcc925727f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:29:32.821037 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.820998    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:32.923553 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:32.923513    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:33.009214 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.009181    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:29:33.080902 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.080872    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75"]
Sep 09 03:29:33.084803 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.084745    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75"
Sep 09 03:29:33.088098 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.087934    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9c982818-dockercfg-rvp9m\""
Sep 09 03:29:33.088412 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.088177    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9c982818\""
Sep 09 03:29:33.098453 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.098421    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75"]
Sep 09 03:29:33.169038 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.169005    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56afff4d-01aa-4b96-8573-e0dcd60fcf36-plugins-config-volume\") pod \"payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75\" (UID: \"56afff4d-01aa-4b96-8573-e0dcd60fcf36\") " pod="openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75"
Sep 09 03:29:33.169211 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.169046    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flbhq\" (UniqueName: \"kubernetes.io/projected/56afff4d-01aa-4b96-8573-e0dcd60fcf36-kube-api-access-flbhq\") pod \"payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75\" (UID: \"56afff4d-01aa-4b96-8573-e0dcd60fcf36\") " pod="openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75"
Sep 09 03:29:33.270256 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.270185    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56afff4d-01aa-4b96-8573-e0dcd60fcf36-plugins-config-volume\") pod \"payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75\" (UID: \"56afff4d-01aa-4b96-8573-e0dcd60fcf36\") " pod="openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75"
Sep 09 03:29:33.270256 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.270227    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flbhq\" (UniqueName: \"kubernetes.io/projected/56afff4d-01aa-4b96-8573-e0dcd60fcf36-kube-api-access-flbhq\") pod \"payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75\" (UID: \"56afff4d-01aa-4b96-8573-e0dcd60fcf36\") " pod="openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75"
Sep 09 03:29:33.270893 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.270872    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56afff4d-01aa-4b96-8573-e0dcd60fcf36-plugins-config-volume\") pod \"payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75\" (UID: \"56afff4d-01aa-4b96-8573-e0dcd60fcf36\") " pod="openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75"
Sep 09 03:29:33.285308 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.285282    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"]
Sep 09 03:29:33.285567 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.285549    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flbhq\" (UniqueName: \"kubernetes.io/projected/56afff4d-01aa-4b96-8573-e0dcd60fcf36-kube-api-access-flbhq\") pod \"payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75\" (UID: \"56afff4d-01aa-4b96-8573-e0dcd60fcf36\") " pod="openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75"
Sep 09 03:29:33.287901 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:29:33.287868    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8082450_ffff_407a_8b28_0efcc925727f.slice/crio-d6ce1078f834e5d7462ebe83a6e153ad0adbac567cb5c6702557ea0373dfedbd WatchSource:0}: Error finding container d6ce1078f834e5d7462ebe83a6e153ad0adbac567cb5c6702557ea0373dfedbd: Status 404 returned error can't find the container with id d6ce1078f834e5d7462ebe83a6e153ad0adbac567cb5c6702557ea0373dfedbd
Sep 09 03:29:33.307343 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.307321    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:29:33.399988 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.399953    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75"
Sep 09 03:29:33.516810 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.516782    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:33.674295 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.674261    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75"]
Sep 09 03:29:33.675392 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:29:33.675365    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56afff4d_01aa_4b96_8573_e0dcd60fcf36.slice/crio-ccda424411f3b0a9c5835d09ad126c838ee738d1720c5134ec1c634f529ce6bc WatchSource:0}: Error finding container ccda424411f3b0a9c5835d09ad126c838ee738d1720c5134ec1c634f529ce6bc: Status 404 returned error can't find the container with id ccda424411f3b0a9c5835d09ad126c838ee738d1720c5134ec1c634f529ce6bc
Sep 09 03:29:33.751718 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.751682    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw" event={"ID":"f8082450-ffff-407a-8b28-0efcc925727f","Type":"ContainerStarted","Data":"d6ce1078f834e5d7462ebe83a6e153ad0adbac567cb5c6702557ea0373dfedbd"}
Sep 09 03:29:33.752828 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:33.752810    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75" event={"ID":"56afff4d-01aa-4b96-8573-e0dcd60fcf36","Type":"ContainerStarted","Data":"ccda424411f3b0a9c5835d09ad126c838ee738d1720c5134ec1c634f529ce6bc"}
Sep 09 03:29:34.415191 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:34.415131    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:35.225438 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:35.225352    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:35.497722 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:35.497621    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:35.664782 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:35.664748    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:35.753733 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:35.753651    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:36.773469 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:36.773429    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw" event={"ID":"f8082450-ffff-407a-8b28-0efcc925727f","Type":"ContainerStarted","Data":"f8ed9ef8a33d266694380f0f91c9a8a844626f3072656f5c3c902214896130d2"}
Sep 09 03:29:36.773957 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:36.773629    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:29:36.775304 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:36.775280    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75" event={"ID":"56afff4d-01aa-4b96-8573-e0dcd60fcf36","Type":"ContainerStarted","Data":"5f2fe223fc2ade33e76a97b523f7528cbdf0881224b0b2b8e873f27505b91277"}
Sep 09 03:29:36.775428 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:36.775402    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75"
Sep 09 03:29:36.801003 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:36.800955    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw" podStartSLOduration=3.163350569 podStartE2EDuration="4.800940397s" totalImagesPullingTime="1.637589828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:29:36.798999399 +0000 UTC m=+993.935747265" watchObservedRunningTime="2026-09-09 03:29:36.800940397 +0000 UTC m=+993.937688261"
Sep 09 03:29:36.831522 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:36.831475    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75" podStartSLOduration=1.525297767 podStartE2EDuration="3.831459359s" totalImagesPullingTime="2.306161592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:29:36.830768779 +0000 UTC m=+993.967516643" watchObservedRunningTime="2026-09-09 03:29:36.831459359 +0000 UTC m=+993.968207224"
Sep 09 03:29:36.948291 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:36.948253    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:37.047281 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:37.047199    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:37.205147 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:37.205111    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:38.404849 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:38.404812    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:40.113903 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:40.113866    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:42.786995 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:42.786963    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:29:43.308563 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:43.308524    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:43.383674 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:43.383638    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:43.478909 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:43.478855    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:43.605268 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:43.605240    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:43.793747 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:43.793690    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:47.782795 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:47.782756    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75"
Sep 09 03:29:48.690258 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:48.690218    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:29:49.514733 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:49.514670    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:49.872142 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:49.872106    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:49.972566 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:49.972530    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:50.071840 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:50.071797    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:50.233701 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:50.233587    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:50.427694 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:50.427649    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:51.854733 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:51.854703    2560 generic.go:356] "Generic (PLEG): container finished" podID="9c65e186-03b1-4ad0-80d6-e2eacf40e324" containerID="6ee71488fcb02ecdc4f17aeb8a5209d2486d3ba72fdf7886c2a737bd974236ea" exitCode=0
Sep 09 03:29:51.855134 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:51.854748    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq" event={"ID":"9c65e186-03b1-4ad0-80d6-e2eacf40e324","Type":"ContainerDied","Data":"6ee71488fcb02ecdc4f17aeb8a5209d2486d3ba72fdf7886c2a737bd974236ea"}
Sep 09 03:29:51.855178 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:51.855132    2560 scope.go:117] "RemoveContainer" containerID="6ee71488fcb02ecdc4f17aeb8a5209d2486d3ba72fdf7886c2a737bd974236ea"
Sep 09 03:29:51.856205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:51.856177    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:29:51.856314 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:51.856258    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:29:51.856314 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:51.856303    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:29:52.219210 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:52.219115    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:52.865875 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:52.865829    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq" event={"ID":"9c65e186-03b1-4ad0-80d6-e2eacf40e324","Type":"ContainerStarted","Data":"ccbf4d8d18fd257618882abf6c85dedaa6916312d237f4f0e7c547888efe6c38"}
Sep 09 03:29:53.488658 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:53.488624    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:53.579704 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:53.578109    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:53.596851 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:53.596811    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:53.597581 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:53.597554    2560 patch_prober.go:28] interesting pod/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 09 03:29:53.597745 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:53.597627    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq" podUID="9c65e186-03b1-4ad0-80d6-e2eacf40e324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 09 03:29:53.678721 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:53.678677    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:54.417662 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:54.417623    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:54.529680 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:54.529648    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:54.594106 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:54.594065    2560 patch_prober.go:28] interesting pod/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 09 03:29:54.594293 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:54.594134    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq" podUID="9c65e186-03b1-4ad0-80d6-e2eacf40e324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 09 03:29:55.533071 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:55.533031    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:55.594558 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:55.594519    2560 patch_prober.go:28] interesting pod/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 09 03:29:55.594765 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:55.594580    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq" podUID="9c65e186-03b1-4ad0-80d6-e2eacf40e324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 09 03:29:56.441229 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.441194    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"]
Sep 09 03:29:56.450580 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.450549    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:29:56.457665 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.455050    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4bd6b8-serving-cert\""
Sep 09 03:29:56.460802 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.460776    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"]
Sep 09 03:29:56.498637 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.498604    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:56.509789 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.509762    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e82c204d-cb79-481e-ade0-255db3330688-metrics-certs\") pod \"maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs\" (UID: \"e82c204d-cb79-481e-ade0-255db3330688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:29:56.509902 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.509800    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82c204d-cb79-481e-ade0-255db3330688-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs\" (UID: \"e82c204d-cb79-481e-ade0-255db3330688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:29:56.509902 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.509829    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82c204d-cb79-481e-ade0-255db3330688-maas-api-tls\") pod \"maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs\" (UID: \"e82c204d-cb79-481e-ade0-255db3330688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:29:56.509902 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.509890    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6hlh\" (UniqueName: \"kubernetes.io/projected/e82c204d-cb79-481e-ade0-255db3330688-kube-api-access-z6hlh\") pod \"maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs\" (UID: \"e82c204d-cb79-481e-ade0-255db3330688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:29:56.593805 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.593769    2560 patch_prober.go:28] interesting pod/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 09 03:29:56.594152 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.593830    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq" podUID="9c65e186-03b1-4ad0-80d6-e2eacf40e324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 09 03:29:56.597261 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.597230    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:56.610708 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.610684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e82c204d-cb79-481e-ade0-255db3330688-metrics-certs\") pod \"maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs\" (UID: \"e82c204d-cb79-481e-ade0-255db3330688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:29:56.610828 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.610728    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82c204d-cb79-481e-ade0-255db3330688-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs\" (UID: \"e82c204d-cb79-481e-ade0-255db3330688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:29:56.610828 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.610770    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82c204d-cb79-481e-ade0-255db3330688-maas-api-tls\") pod \"maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs\" (UID: \"e82c204d-cb79-481e-ade0-255db3330688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:29:56.610828 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.610794    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6hlh\" (UniqueName: \"kubernetes.io/projected/e82c204d-cb79-481e-ade0-255db3330688-kube-api-access-z6hlh\") pod \"maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs\" (UID: \"e82c204d-cb79-481e-ade0-255db3330688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:29:56.610951 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:29:56.610912    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-serving-cert: secret "maas-api-e2e-inf-b-4bd6b8-serving-cert" not found
Sep 09 03:29:56.610995 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:29:56.610987    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e82c204d-cb79-481e-ade0-255db3330688-maas-api-tls podName:e82c204d-cb79-481e-ade0-255db3330688 nodeName:}" failed. No retries permitted until 2026-09-09 03:29:57.110967171 +0000 UTC m=+1014.247715028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e82c204d-cb79-481e-ade0-255db3330688-maas-api-tls") pod "maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs" (UID: "e82c204d-cb79-481e-ade0-255db3330688") : secret "maas-api-e2e-inf-b-4bd6b8-serving-cert" not found
Sep 09 03:29:56.611503 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.611479    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82c204d-cb79-481e-ade0-255db3330688-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs\" (UID: \"e82c204d-cb79-481e-ade0-255db3330688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:29:56.613097 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.613066    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e82c204d-cb79-481e-ade0-255db3330688-metrics-certs\") pod \"maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs\" (UID: \"e82c204d-cb79-481e-ade0-255db3330688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:29:56.619999 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.619981    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6hlh\" (UniqueName: \"kubernetes.io/projected/e82c204d-cb79-481e-ade0-255db3330688-kube-api-access-z6hlh\") pod \"maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs\" (UID: \"e82c204d-cb79-481e-ade0-255db3330688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:29:56.883290 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:56.883257    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:57.015779 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:57.015745    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:57.115970 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:57.115926    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82c204d-cb79-481e-ade0-255db3330688-maas-api-tls\") pod \"maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs\" (UID: \"e82c204d-cb79-481e-ade0-255db3330688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:29:57.118316 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:57.118288    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82c204d-cb79-481e-ade0-255db3330688-maas-api-tls\") pod \"maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs\" (UID: \"e82c204d-cb79-481e-ade0-255db3330688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:29:57.141994 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:57.141911    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:57.378467 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:57.378433    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:29:57.594267 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:57.594228    2560 patch_prober.go:28] interesting pod/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 09 03:29:57.594700 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:57.594279    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq" podUID="9c65e186-03b1-4ad0-80d6-e2eacf40e324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 09 03:29:57.672354 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:57.672251    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"]
Sep 09 03:29:57.680643 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:29:57.680587    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82c204d_cb79_481e_ade0_255db3330688.slice/crio-2fcffe4e6390f1a2c10b3318148b4cedb478ca8ce5b0d3b717ecbc2f1de80972 WatchSource:0}: Error finding container 2fcffe4e6390f1a2c10b3318148b4cedb478ca8ce5b0d3b717ecbc2f1de80972: Status 404 returned error can't find the container with id 2fcffe4e6390f1a2c10b3318148b4cedb478ca8ce5b0d3b717ecbc2f1de80972
Sep 09 03:29:57.845797 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:57.845761    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:57.899150 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:57.899114    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs" event={"ID":"e82c204d-cb79-481e-ade0-255db3330688","Type":"ContainerStarted","Data":"2fcffe4e6390f1a2c10b3318148b4cedb478ca8ce5b0d3b717ecbc2f1de80972"}
Sep 09 03:29:58.597179 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:58.597148    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:58.597678 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:58.597532    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:58.598065 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:58.598043    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:29:58.906149 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:58.906113    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs" event={"ID":"e82c204d-cb79-481e-ade0-255db3330688","Type":"ContainerStarted","Data":"5e2c6993478ab9e08572f8533278412003f9f34013c8afaa95d66faf524d4700"}
Sep 09 03:29:58.906349 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:58.906206    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:29:58.914890 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:58.914859    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:29:58.927247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:29:58.927199    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs" podStartSLOduration=2.589145556 podStartE2EDuration="2.927183847s" totalImagesPullingTime="338.038291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:29:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:29:58.925578561 +0000 UTC m=+1016.062326426" watchObservedRunningTime="2026-09-09 03:29:58.927183847 +0000 UTC m=+1016.063931755"
Sep 09 03:30:00.141708 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:00.141672    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2"]
Sep 09 03:30:00.145445 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:00.145417    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2"
Sep 09 03:30:00.147749 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:00.147714    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-a2fc47-29815410-qm65d"]
Sep 09 03:30:00.151053 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:00.151033    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-a2fc47-29815410-qm65d"
Sep 09 03:30:00.151311 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:00.151293    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7q9f\""
Sep 09 03:30:00.155370 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:00.155355    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2"
Sep 09 03:30:00.160577 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:00.160554    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2"]
Sep 09 03:30:00.164838 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:00.164821    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-a2fc47-29815410-qm65d"
Sep 09 03:30:00.176355 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:00.176331    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-a2fc47-29815410-qm65d"]
Sep 09 03:30:00.439969 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:00.439945    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2"]
Sep 09 03:30:00.441841 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:30:00.441809    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c9f45fa_75fb_4fd6_b68a_138e05c6f672.slice/crio-650a6d020cc26ef92d6c9ceaf0fdc23d308190ecc43224dee3184e7b7c4ce7fb WatchSource:0}: Error finding container 650a6d020cc26ef92d6c9ceaf0fdc23d308190ecc43224dee3184e7b7c4ce7fb: Status 404 returned error can't find the container with id 650a6d020cc26ef92d6c9ceaf0fdc23d308190ecc43224dee3184e7b7c4ce7fb
Sep 09 03:30:00.450989 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:00.450964    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-a2fc47-29815410-qm65d"]
Sep 09 03:30:00.452835 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:30:00.452811    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb309424_3aad_447e_8700_0da729a89d17.slice/crio-86f2b57b445abeb9a83f8ed00bbe0990d8c66fd353192ecaf194db06a410f010 WatchSource:0}: Error finding container 86f2b57b445abeb9a83f8ed00bbe0990d8c66fd353192ecaf194db06a410f010: Status 404 returned error can't find the container with id 86f2b57b445abeb9a83f8ed00bbe0990d8c66fd353192ecaf194db06a410f010
Sep 09 03:30:00.916739 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:00.916704    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-a2fc47-29815410-qm65d" event={"ID":"db309424-3aad-447e-8700-0da729a89d17","Type":"ContainerStarted","Data":"86f2b57b445abeb9a83f8ed00bbe0990d8c66fd353192ecaf194db06a410f010"}
Sep 09 03:30:00.917873 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:00.917852    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2" event={"ID":"8c9f45fa-75fb-4fd6-b68a-138e05c6f672","Type":"ContainerStarted","Data":"650a6d020cc26ef92d6c9ceaf0fdc23d308190ecc43224dee3184e7b7c4ce7fb"}
Sep 09 03:30:02.932500 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:02.932466    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2" event={"ID":"8c9f45fa-75fb-4fd6-b68a-138e05c6f672","Type":"ContainerStarted","Data":"689faeba18a11df68da6d9b69d0d175bcf6aef494936aa3cc67710cfaa5c92ff"}
Sep 09 03:30:02.934017 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:02.933991    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-a2fc47-29815410-qm65d" event={"ID":"db309424-3aad-447e-8700-0da729a89d17","Type":"ContainerStarted","Data":"764e39c0e77adee4d9d0967f8f1ceadc2b4466010adab323e6668d0c39e05f93"}
Sep 09 03:30:02.950666 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:02.950574    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2" podStartSLOduration=0.81082081 podStartE2EDuration="2.95055704s" totalImagesPullingTime="2.13973623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:30:02.949864225 +0000 UTC m=+1020.086612090" watchObservedRunningTime="2026-09-09 03:30:02.95055704 +0000 UTC m=+1020.087304904"
Sep 09 03:30:02.970717 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:02.970659    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-a2fc47-29815410-qm65d" podStartSLOduration=0.846032099 podStartE2EDuration="2.970641234s" totalImagesPullingTime="2.124609135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:30:02.968144175 +0000 UTC m=+1020.104892041" watchObservedRunningTime="2026-09-09 03:30:02.970641234 +0000 UTC m=+1020.107389111"
Sep 09 03:30:03.940861 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:03.940823    2560 generic.go:356] "Generic (PLEG): container finished" podID="8c9f45fa-75fb-4fd6-b68a-138e05c6f672" containerID="689faeba18a11df68da6d9b69d0d175bcf6aef494936aa3cc67710cfaa5c92ff" exitCode=7
Sep 09 03:30:03.941290 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:03.940921    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2" event={"ID":"8c9f45fa-75fb-4fd6-b68a-138e05c6f672","Type":"ContainerDied","Data":"689faeba18a11df68da6d9b69d0d175bcf6aef494936aa3cc67710cfaa5c92ff"}
Sep 09 03:30:03.941290 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:03.941058    2560 scope.go:117] "RemoveContainer" containerID="689faeba18a11df68da6d9b69d0d175bcf6aef494936aa3cc67710cfaa5c92ff"
Sep 09 03:30:03.942426 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:03.942400    2560 generic.go:356] "Generic (PLEG): container finished" podID="db309424-3aad-447e-8700-0da729a89d17" containerID="764e39c0e77adee4d9d0967f8f1ceadc2b4466010adab323e6668d0c39e05f93" exitCode=0
Sep 09 03:30:03.942545 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:03.942526    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-a2fc47-29815410-qm65d" event={"ID":"db309424-3aad-447e-8700-0da729a89d17","Type":"ContainerDied","Data":"764e39c0e77adee4d9d0967f8f1ceadc2b4466010adab323e6668d0c39e05f93"}
Sep 09 03:30:04.916524 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:04.916494    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:30:04.950978 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:04.950946    2560 generic.go:356] "Generic (PLEG): container finished" podID="8c9f45fa-75fb-4fd6-b68a-138e05c6f672" containerID="89b56ab89d4fe362044a153c7abce2279cfb3d10ad4ff71c0a0d6d67e71f05d0" exitCode=7
Sep 09 03:30:04.951455 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:04.951032    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2" event={"ID":"8c9f45fa-75fb-4fd6-b68a-138e05c6f672","Type":"ContainerDied","Data":"89b56ab89d4fe362044a153c7abce2279cfb3d10ad4ff71c0a0d6d67e71f05d0"}
Sep 09 03:30:04.951455 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:04.951104    2560 scope.go:117] "RemoveContainer" containerID="689faeba18a11df68da6d9b69d0d175bcf6aef494936aa3cc67710cfaa5c92ff"
Sep 09 03:30:04.951455 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:04.951242    2560 scope.go:117] "RemoveContainer" containerID="89b56ab89d4fe362044a153c7abce2279cfb3d10ad4ff71c0a0d6d67e71f05d0"
Sep 09 03:30:04.951650 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:30:04.951519    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2_odh-ai-gateway-infra(8c9f45fa-75fb-4fd6-b68a-138e05c6f672)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2" podUID="8c9f45fa-75fb-4fd6-b68a-138e05c6f672"
Sep 09 03:30:05.097617 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:05.097576    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-a2fc47-29815410-qm65d"
Sep 09 03:30:05.958499 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:05.958470    2560 scope.go:117] "RemoveContainer" containerID="89b56ab89d4fe362044a153c7abce2279cfb3d10ad4ff71c0a0d6d67e71f05d0"
Sep 09 03:30:05.958994 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:30:05.958744    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2_odh-ai-gateway-infra(8c9f45fa-75fb-4fd6-b68a-138e05c6f672)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2" podUID="8c9f45fa-75fb-4fd6-b68a-138e05c6f672"
Sep 09 03:30:05.959569 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:05.959554    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-a2fc47-29815410-qm65d"
Sep 09 03:30:05.959681 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:05.959582    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-a2fc47-29815410-qm65d" event={"ID":"db309424-3aad-447e-8700-0da729a89d17","Type":"ContainerDied","Data":"86f2b57b445abeb9a83f8ed00bbe0990d8c66fd353192ecaf194db06a410f010"}
Sep 09 03:30:05.959681 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:05.959648    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86f2b57b445abeb9a83f8ed00bbe0990d8c66fd353192ecaf194db06a410f010"
Sep 09 03:30:06.300478 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:06.300389    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:30:11.651058 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:11.651020    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:12.248513 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.248473    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"]
Sep 09 03:30:12.248793 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.248735    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw" podUID="f8082450-ffff-407a-8b28-0efcc925727f" containerName="maas-api" containerID="cri-o://f8ed9ef8a33d266694380f0f91c9a8a844626f3072656f5c3c902214896130d2" gracePeriod=30
Sep 09 03:30:12.282487 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.282456    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:12.309378 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.309352    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75"]
Sep 09 03:30:12.309562 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.309539    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75" podUID="56afff4d-01aa-4b96-8573-e0dcd60fcf36" containerName="payload-processing" containerID="cri-o://5f2fe223fc2ade33e76a97b523f7528cbdf0881224b0b2b8e873f27505b91277" gracePeriod=30
Sep 09 03:30:12.385216 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.385183    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:12.480776 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.480541    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:12.556698 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.556675    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:30:12.582557 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.582533    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75"
Sep 09 03:30:12.585182 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.585141    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:12.653090 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.653061    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"]
Sep 09 03:30:12.653464 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.653245    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq" podUID="9c65e186-03b1-4ad0-80d6-e2eacf40e324" containerName="istio-proxy" containerID="cri-o://ccbf4d8d18fd257618882abf6c85dedaa6916312d237f4f0e7c547888efe6c38" gracePeriod=30
Sep 09 03:30:12.665834 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.665807    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8082450-ffff-407a-8b28-0efcc925727f-maas-api-tls\") pod \"f8082450-ffff-407a-8b28-0efcc925727f\" (UID: \"f8082450-ffff-407a-8b28-0efcc925727f\") "
Sep 09 03:30:12.665961 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.665876    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg5mh\" (UniqueName: \"kubernetes.io/projected/f8082450-ffff-407a-8b28-0efcc925727f-kube-api-access-gg5mh\") pod \"f8082450-ffff-407a-8b28-0efcc925727f\" (UID: \"f8082450-ffff-407a-8b28-0efcc925727f\") "
Sep 09 03:30:12.665961 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.665908    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56afff4d-01aa-4b96-8573-e0dcd60fcf36-plugins-config-volume\") pod \"56afff4d-01aa-4b96-8573-e0dcd60fcf36\" (UID: \"56afff4d-01aa-4b96-8573-e0dcd60fcf36\") "
Sep 09 03:30:12.665961 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.665955    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8082450-ffff-407a-8b28-0efcc925727f-openshift-service-ca\") pod \"f8082450-ffff-407a-8b28-0efcc925727f\" (UID: \"f8082450-ffff-407a-8b28-0efcc925727f\") "
Sep 09 03:30:12.666124 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.666045    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8082450-ffff-407a-8b28-0efcc925727f-metrics-certs\") pod \"f8082450-ffff-407a-8b28-0efcc925727f\" (UID: \"f8082450-ffff-407a-8b28-0efcc925727f\") "
Sep 09 03:30:12.666124 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.666075    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flbhq\" (UniqueName: \"kubernetes.io/projected/56afff4d-01aa-4b96-8573-e0dcd60fcf36-kube-api-access-flbhq\") pod \"56afff4d-01aa-4b96-8573-e0dcd60fcf36\" (UID: \"56afff4d-01aa-4b96-8573-e0dcd60fcf36\") "
Sep 09 03:30:12.666312 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.666277    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8082450-ffff-407a-8b28-0efcc925727f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f8082450-ffff-407a-8b28-0efcc925727f" (UID: "f8082450-ffff-407a-8b28-0efcc925727f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:30:12.666401 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.666349    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56afff4d-01aa-4b96-8573-e0dcd60fcf36-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56afff4d-01aa-4b96-8573-e0dcd60fcf36" (UID: "56afff4d-01aa-4b96-8573-e0dcd60fcf36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:30:12.666552 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.666533    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56afff4d-01aa-4b96-8573-e0dcd60fcf36-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:12.666644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.666561    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8082450-ffff-407a-8b28-0efcc925727f-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:12.668006 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.667984    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8082450-ffff-407a-8b28-0efcc925727f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f8082450-ffff-407a-8b28-0efcc925727f" (UID: "f8082450-ffff-407a-8b28-0efcc925727f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:30:12.668120 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.668102    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8082450-ffff-407a-8b28-0efcc925727f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f8082450-ffff-407a-8b28-0efcc925727f" (UID: "f8082450-ffff-407a-8b28-0efcc925727f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:30:12.668188 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.668166    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8082450-ffff-407a-8b28-0efcc925727f-kube-api-access-gg5mh" (OuterVolumeSpecName: "kube-api-access-gg5mh") pod "f8082450-ffff-407a-8b28-0efcc925727f" (UID: "f8082450-ffff-407a-8b28-0efcc925727f"). InnerVolumeSpecName "kube-api-access-gg5mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:30:12.668372 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.668358    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56afff4d-01aa-4b96-8573-e0dcd60fcf36-kube-api-access-flbhq" (OuterVolumeSpecName: "kube-api-access-flbhq") pod "56afff4d-01aa-4b96-8573-e0dcd60fcf36" (UID: "56afff4d-01aa-4b96-8573-e0dcd60fcf36"). InnerVolumeSpecName "kube-api-access-flbhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:30:12.692428 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.692398    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:12.767864 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.767768    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8082450-ffff-407a-8b28-0efcc925727f-metrics-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:12.767864 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.767799    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flbhq\" (UniqueName: \"kubernetes.io/projected/56afff4d-01aa-4b96-8573-e0dcd60fcf36-kube-api-access-flbhq\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:12.767864 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.767814    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8082450-ffff-407a-8b28-0efcc925727f-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:12.767864 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.767827    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg5mh\" (UniqueName: \"kubernetes.io/projected/f8082450-ffff-407a-8b28-0efcc925727f-kube-api-access-gg5mh\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:12.996543 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.996507    2560 generic.go:356] "Generic (PLEG): container finished" podID="f8082450-ffff-407a-8b28-0efcc925727f" containerID="f8ed9ef8a33d266694380f0f91c9a8a844626f3072656f5c3c902214896130d2" exitCode=0
Sep 09 03:30:12.996753 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.996557    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"
Sep 09 03:30:12.996753 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.996580    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw" event={"ID":"f8082450-ffff-407a-8b28-0efcc925727f","Type":"ContainerDied","Data":"f8ed9ef8a33d266694380f0f91c9a8a844626f3072656f5c3c902214896130d2"}
Sep 09 03:30:12.996753 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.996672    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw" event={"ID":"f8082450-ffff-407a-8b28-0efcc925727f","Type":"ContainerDied","Data":"d6ce1078f834e5d7462ebe83a6e153ad0adbac567cb5c6702557ea0373dfedbd"}
Sep 09 03:30:12.996753 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.996704    2560 scope.go:117] "RemoveContainer" containerID="f8ed9ef8a33d266694380f0f91c9a8a844626f3072656f5c3c902214896130d2"
Sep 09 03:30:12.997982 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.997961    2560 generic.go:356] "Generic (PLEG): container finished" podID="56afff4d-01aa-4b96-8573-e0dcd60fcf36" containerID="5f2fe223fc2ade33e76a97b523f7528cbdf0881224b0b2b8e873f27505b91277" exitCode=0
Sep 09 03:30:12.998055 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.998002    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75" event={"ID":"56afff4d-01aa-4b96-8573-e0dcd60fcf36","Type":"ContainerDied","Data":"5f2fe223fc2ade33e76a97b523f7528cbdf0881224b0b2b8e873f27505b91277"}
Sep 09 03:30:12.998055 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.998012    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75"
Sep 09 03:30:12.998055 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:12.998034    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75" event={"ID":"56afff4d-01aa-4b96-8573-e0dcd60fcf36","Type":"ContainerDied","Data":"ccda424411f3b0a9c5835d09ad126c838ee738d1720c5134ec1c634f529ce6bc"}
Sep 09 03:30:13.006429 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.006410    2560 scope.go:117] "RemoveContainer" containerID="f8ed9ef8a33d266694380f0f91c9a8a844626f3072656f5c3c902214896130d2"
Sep 09 03:30:13.006725 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:30:13.006700    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ed9ef8a33d266694380f0f91c9a8a844626f3072656f5c3c902214896130d2\": container with ID starting with f8ed9ef8a33d266694380f0f91c9a8a844626f3072656f5c3c902214896130d2 not found: ID does not exist" containerID="f8ed9ef8a33d266694380f0f91c9a8a844626f3072656f5c3c902214896130d2"
Sep 09 03:30:13.006791 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.006736    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ed9ef8a33d266694380f0f91c9a8a844626f3072656f5c3c902214896130d2"} err="failed to get container status \"f8ed9ef8a33d266694380f0f91c9a8a844626f3072656f5c3c902214896130d2\": rpc error: code = NotFound desc = could not find container \"f8ed9ef8a33d266694380f0f91c9a8a844626f3072656f5c3c902214896130d2\": container with ID starting with f8ed9ef8a33d266694380f0f91c9a8a844626f3072656f5c3c902214896130d2 not found: ID does not exist"
Sep 09 03:30:13.006791 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.006762    2560 scope.go:117] "RemoveContainer" containerID="5f2fe223fc2ade33e76a97b523f7528cbdf0881224b0b2b8e873f27505b91277"
Sep 09 03:30:13.015065 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.015049    2560 scope.go:117] "RemoveContainer" containerID="5f2fe223fc2ade33e76a97b523f7528cbdf0881224b0b2b8e873f27505b91277"
Sep 09 03:30:13.015301 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:30:13.015286    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f2fe223fc2ade33e76a97b523f7528cbdf0881224b0b2b8e873f27505b91277\": container with ID starting with 5f2fe223fc2ade33e76a97b523f7528cbdf0881224b0b2b8e873f27505b91277 not found: ID does not exist" containerID="5f2fe223fc2ade33e76a97b523f7528cbdf0881224b0b2b8e873f27505b91277"
Sep 09 03:30:13.015342 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.015308    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f2fe223fc2ade33e76a97b523f7528cbdf0881224b0b2b8e873f27505b91277"} err="failed to get container status \"5f2fe223fc2ade33e76a97b523f7528cbdf0881224b0b2b8e873f27505b91277\": rpc error: code = NotFound desc = could not find container \"5f2fe223fc2ade33e76a97b523f7528cbdf0881224b0b2b8e873f27505b91277\": container with ID starting with 5f2fe223fc2ade33e76a97b523f7528cbdf0881224b0b2b8e873f27505b91277 not found: ID does not exist"
Sep 09 03:30:13.024174 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.024125    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"]
Sep 09 03:30:13.027822 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.027799    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c982818-6657f4cf57-662kw"]
Sep 09 03:30:13.043402 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.043375    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75"]
Sep 09 03:30:13.047970 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.047952    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9c982818-6db6664cdc-q7t75"]
Sep 09 03:30:13.229627 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.229585    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:13.315509 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.315430    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:13.358435 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.358401    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56afff4d-01aa-4b96-8573-e0dcd60fcf36" path="/var/lib/kubelet/pods/56afff4d-01aa-4b96-8573-e0dcd60fcf36/volumes"
Sep 09 03:30:13.358829 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.358814    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8082450-ffff-407a-8b28-0efcc925727f" path="/var/lib/kubelet/pods/f8082450-ffff-407a-8b28-0efcc925727f/volumes"
Sep 09 03:30:13.468227 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.468194    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:13.599801 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.599716    2560 patch_prober.go:28] interesting pod/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 03:30:13.600059 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.599803    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq" podUID="9c65e186-03b1-4ad0-80d6-e2eacf40e324" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 03:30:13.727713 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.727677    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"]
Sep 09 03:30:13.728143 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.728128    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56afff4d-01aa-4b96-8573-e0dcd60fcf36" containerName="payload-processing"
Sep 09 03:30:13.728143 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.728145    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="56afff4d-01aa-4b96-8573-e0dcd60fcf36" containerName="payload-processing"
Sep 09 03:30:13.728233 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.728153    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db309424-3aad-447e-8700-0da729a89d17" containerName="cleanup"
Sep 09 03:30:13.728233 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.728159    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="db309424-3aad-447e-8700-0da729a89d17" containerName="cleanup"
Sep 09 03:30:13.728233 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.728173    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8082450-ffff-407a-8b28-0efcc925727f" containerName="maas-api"
Sep 09 03:30:13.728233 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.728182    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8082450-ffff-407a-8b28-0efcc925727f" containerName="maas-api"
Sep 09 03:30:13.728390 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.728245    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8082450-ffff-407a-8b28-0efcc925727f" containerName="maas-api"
Sep 09 03:30:13.728390 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.728255    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="db309424-3aad-447e-8700-0da729a89d17" containerName="cleanup"
Sep 09 03:30:13.728390 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.728268    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="56afff4d-01aa-4b96-8573-e0dcd60fcf36" containerName="payload-processing"
Sep 09 03:30:13.734555 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.734534    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.737771 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.737750    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-5f9aae59-gw-openshift-default-dockercfg-2p676\""
Sep 09 03:30:13.745654 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.745631    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"]
Sep 09 03:30:13.762079 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.762047    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:13.879781 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.879747    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwfck\" (UniqueName: \"kubernetes.io/projected/a3f31d25-6baa-48ce-ae38-4f8709bc0687-kube-api-access-xwfck\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.879954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.879787    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-token\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.879954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.879843    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-data\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.879954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.879885    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-podinfo\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.880063 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.879967    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istiod-ca-cert\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.880063 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.880012    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-envoy\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.880063 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.880046    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-credential-socket\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.880162 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.880089    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-workload-certs\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.880162 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.880129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-workload-socket\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.981296 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.981267    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-credential-socket\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.981494 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.981312    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-workload-certs\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.981494 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.981339    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-workload-socket\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.981494 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.981373    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwfck\" (UniqueName: \"kubernetes.io/projected/a3f31d25-6baa-48ce-ae38-4f8709bc0687-kube-api-access-xwfck\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.981494 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.981407    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-token\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.981736 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.981622    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-data\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.981736 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.981657    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-podinfo\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.981736 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.981730    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istiod-ca-cert\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.981891 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.981771    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-credential-socket\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.981891 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.981807    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-envoy\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.982004 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.981932    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-workload-socket\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.982004 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.981966    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-workload-certs\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.982255 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.982236    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-data\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.982705 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.982685    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istiod-ca-cert\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.984621 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.984586    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-envoy\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.984740 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.984725    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-podinfo\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.993383 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.993360    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-token\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:13.993511 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:13.993490    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwfck\" (UniqueName: \"kubernetes.io/projected/a3f31d25-6baa-48ce-ae38-4f8709bc0687-kube-api-access-xwfck\") pod \"e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") " pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:14.046496 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:14.046469    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:14.063204 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:14.063166    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:14.322619 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:14.322571    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"]
Sep 09 03:30:14.324622 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:30:14.324585    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f31d25_6baa_48ce_ae38_4f8709bc0687.slice/crio-3c0d4434b9a2fd81967b61a4949979b99bb76ebf79b14f9b59ba795506e9ba29 WatchSource:0}: Error finding container 3c0d4434b9a2fd81967b61a4949979b99bb76ebf79b14f9b59ba795506e9ba29: Status 404 returned error can't find the container with id 3c0d4434b9a2fd81967b61a4949979b99bb76ebf79b14f9b59ba795506e9ba29
Sep 09 03:30:14.327303 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:14.327271    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:30:14.327401 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:14.327348    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:30:14.327401 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:14.327391    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:30:15.013885 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:15.013853    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg" event={"ID":"a3f31d25-6baa-48ce-ae38-4f8709bc0687","Type":"ContainerStarted","Data":"fad9557d38644b25b1434514cfc2e6afeb0024b9c0c0b4bc9872b90ce6b68566"}
Sep 09 03:30:15.014269 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:15.013911    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg" event={"ID":"a3f31d25-6baa-48ce-ae38-4f8709bc0687","Type":"ContainerStarted","Data":"3c0d4434b9a2fd81967b61a4949979b99bb76ebf79b14f9b59ba795506e9ba29"}
Sep 09 03:30:15.036986 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:15.036934    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg" podStartSLOduration=2.036917251 podStartE2EDuration="2.036917251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:30:15.036305184 +0000 UTC m=+1032.173053049" watchObservedRunningTime="2026-09-09 03:30:15.036917251 +0000 UTC m=+1032.173665117"
Sep 09 03:30:15.046880 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:15.046852    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:15.051722 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:15.051700    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:16.018495 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.018451    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:16.019375 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.019356    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:16.526167 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.526133    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:16.627292 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.627250    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:16.631588 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.631558    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"]
Sep 09 03:30:16.635434 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.635413    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:16.639366 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.639346    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-5f9aae59-serving-cert\""
Sep 09 03:30:16.649095 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.649071    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"]
Sep 09 03:30:16.725642 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.725607    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:16.815286 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.815202    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fada035-7396-4acc-94fd-fab40bfaee87-maas-api-tls\") pod \"maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8\" (UID: \"4fada035-7396-4acc-94fd-fab40bfaee87\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:16.815286 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.815278    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfkln\" (UniqueName: \"kubernetes.io/projected/4fada035-7396-4acc-94fd-fab40bfaee87-kube-api-access-zfkln\") pod \"maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8\" (UID: \"4fada035-7396-4acc-94fd-fab40bfaee87\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:16.815490 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.815348    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fada035-7396-4acc-94fd-fab40bfaee87-openshift-service-ca\") pod \"maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8\" (UID: \"4fada035-7396-4acc-94fd-fab40bfaee87\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:16.815490 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.815410    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fada035-7396-4acc-94fd-fab40bfaee87-metrics-certs\") pod \"maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8\" (UID: \"4fada035-7396-4acc-94fd-fab40bfaee87\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:16.916953 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.916923    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fada035-7396-4acc-94fd-fab40bfaee87-metrics-certs\") pod \"maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8\" (UID: \"4fada035-7396-4acc-94fd-fab40bfaee87\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:16.917158 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.916980    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fada035-7396-4acc-94fd-fab40bfaee87-maas-api-tls\") pod \"maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8\" (UID: \"4fada035-7396-4acc-94fd-fab40bfaee87\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:16.917158 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.917079    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfkln\" (UniqueName: \"kubernetes.io/projected/4fada035-7396-4acc-94fd-fab40bfaee87-kube-api-access-zfkln\") pod \"maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8\" (UID: \"4fada035-7396-4acc-94fd-fab40bfaee87\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:16.917158 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.917123    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fada035-7396-4acc-94fd-fab40bfaee87-openshift-service-ca\") pod \"maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8\" (UID: \"4fada035-7396-4acc-94fd-fab40bfaee87\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:16.917158 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:30:16.917136    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-serving-cert: secret "maas-api-e2e-migrate-5f9aae59-serving-cert" not found
Sep 09 03:30:16.917378 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:30:16.917206    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fada035-7396-4acc-94fd-fab40bfaee87-maas-api-tls podName:4fada035-7396-4acc-94fd-fab40bfaee87 nodeName:}" failed. No retries permitted until 2026-09-09 03:30:17.417189562 +0000 UTC m=+1034.553937418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4fada035-7396-4acc-94fd-fab40bfaee87-maas-api-tls") pod "maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8" (UID: "4fada035-7396-4acc-94fd-fab40bfaee87") : secret "maas-api-e2e-migrate-5f9aae59-serving-cert" not found
Sep 09 03:30:16.917776 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.917754    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fada035-7396-4acc-94fd-fab40bfaee87-openshift-service-ca\") pod \"maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8\" (UID: \"4fada035-7396-4acc-94fd-fab40bfaee87\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:16.919481 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.919461    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fada035-7396-4acc-94fd-fab40bfaee87-metrics-certs\") pod \"maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8\" (UID: \"4fada035-7396-4acc-94fd-fab40bfaee87\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:16.926057 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:16.926036    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfkln\" (UniqueName: \"kubernetes.io/projected/4fada035-7396-4acc-94fd-fab40bfaee87-kube-api-access-zfkln\") pod \"maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8\" (UID: \"4fada035-7396-4acc-94fd-fab40bfaee87\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:17.324394 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:17.324361    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:17.422539 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:17.422501    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fada035-7396-4acc-94fd-fab40bfaee87-maas-api-tls\") pod \"maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8\" (UID: \"4fada035-7396-4acc-94fd-fab40bfaee87\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:17.424899 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:17.424878    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fada035-7396-4acc-94fd-fab40bfaee87-maas-api-tls\") pod \"maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8\" (UID: \"4fada035-7396-4acc-94fd-fab40bfaee87\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:17.457872 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:17.457845    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:17.547570 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:17.547533    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:17.605158 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:17.605124    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:17.830455 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:17.830428    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"]
Sep 09 03:30:17.831435 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:30:17.831405    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fada035_7396_4acc_94fd_fab40bfaee87.slice/crio-878681d3102c0f7bed2f05198eaad296536f9577a83b1c108ebb232f30e4be94 WatchSource:0}: Error finding container 878681d3102c0f7bed2f05198eaad296536f9577a83b1c108ebb232f30e4be94: Status 404 returned error can't find the container with id 878681d3102c0f7bed2f05198eaad296536f9577a83b1c108ebb232f30e4be94
Sep 09 03:30:17.905875 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:17.905849    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:30:18.028316 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.028282    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-podinfo\") pod \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") "
Sep 09 03:30:18.028515 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.028336    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v724\" (UniqueName: \"kubernetes.io/projected/9c65e186-03b1-4ad0-80d6-e2eacf40e324-kube-api-access-5v724\") pod \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") "
Sep 09 03:30:18.028515 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.028362    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-data\") pod \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") "
Sep 09 03:30:18.028669 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.028634    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c65e186-03b1-4ad0-80d6-e2eacf40e324" (UID: "9c65e186-03b1-4ad0-80d6-e2eacf40e324"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:30:18.028732 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.028671    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istiod-ca-cert\") pod \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") "
Sep 09 03:30:18.028732 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.028721    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-token\") pod \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") "
Sep 09 03:30:18.028846 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.028821    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-workload-socket\") pod \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") "
Sep 09 03:30:18.028900 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.028867    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-envoy\") pod \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") "
Sep 09 03:30:18.028957 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.028900    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-workload-certs\") pod \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") "
Sep 09 03:30:18.029009 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.028962    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-credential-socket\") pod \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\" (UID: \"9c65e186-03b1-4ad0-80d6-e2eacf40e324\") "
Sep 09 03:30:18.029009 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.028993    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c65e186-03b1-4ad0-80d6-e2eacf40e324" (UID: "9c65e186-03b1-4ad0-80d6-e2eacf40e324"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:30:18.029446 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.029190    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c65e186-03b1-4ad0-80d6-e2eacf40e324" (UID: "9c65e186-03b1-4ad0-80d6-e2eacf40e324"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:30:18.029446 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.029244    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c65e186-03b1-4ad0-80d6-e2eacf40e324" (UID: "9c65e186-03b1-4ad0-80d6-e2eacf40e324"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:30:18.029446 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.029316    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-workload-socket\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:18.029446 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.029337    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-workload-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:18.029446 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.029409    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-data\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:18.029446 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.029412    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c65e186-03b1-4ad0-80d6-e2eacf40e324" (UID: "9c65e186-03b1-4ad0-80d6-e2eacf40e324"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:30:18.029446 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.029422    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istiod-ca-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:18.031049 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.031024    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c65e186-03b1-4ad0-80d6-e2eacf40e324" (UID: "9c65e186-03b1-4ad0-80d6-e2eacf40e324"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:30:18.031175 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.031141    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c65e186-03b1-4ad0-80d6-e2eacf40e324-kube-api-access-5v724" (OuterVolumeSpecName: "kube-api-access-5v724") pod "9c65e186-03b1-4ad0-80d6-e2eacf40e324" (UID: "9c65e186-03b1-4ad0-80d6-e2eacf40e324"). InnerVolumeSpecName "kube-api-access-5v724". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:30:18.031431 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.031407    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c65e186-03b1-4ad0-80d6-e2eacf40e324" (UID: "9c65e186-03b1-4ad0-80d6-e2eacf40e324"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:30:18.031740 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.031721    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c65e186-03b1-4ad0-80d6-e2eacf40e324" (UID: "9c65e186-03b1-4ad0-80d6-e2eacf40e324"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:30:18.034312 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.034280    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8" event={"ID":"4fada035-7396-4acc-94fd-fab40bfaee87","Type":"ContainerStarted","Data":"878681d3102c0f7bed2f05198eaad296536f9577a83b1c108ebb232f30e4be94"}
Sep 09 03:30:18.036476 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.036452    2560 generic.go:356] "Generic (PLEG): container finished" podID="9c65e186-03b1-4ad0-80d6-e2eacf40e324" containerID="ccbf4d8d18fd257618882abf6c85dedaa6916312d237f4f0e7c547888efe6c38" exitCode=0
Sep 09 03:30:18.036561 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.036491    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"
Sep 09 03:30:18.036561 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.036519    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq" event={"ID":"9c65e186-03b1-4ad0-80d6-e2eacf40e324","Type":"ContainerDied","Data":"ccbf4d8d18fd257618882abf6c85dedaa6916312d237f4f0e7c547888efe6c38"}
Sep 09 03:30:18.036561 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.036555    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq" event={"ID":"9c65e186-03b1-4ad0-80d6-e2eacf40e324","Type":"ContainerDied","Data":"f79b9ecf6cd722626d05728dab9d54b7f44aaeed60c945d26e9c576e1388f9fb"}
Sep 09 03:30:18.036755 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.036582    2560 scope.go:117] "RemoveContainer" containerID="ccbf4d8d18fd257618882abf6c85dedaa6916312d237f4f0e7c547888efe6c38"
Sep 09 03:30:18.056737 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.056717    2560 scope.go:117] "RemoveContainer" containerID="6ee71488fcb02ecdc4f17aeb8a5209d2486d3ba72fdf7886c2a737bd974236ea"
Sep 09 03:30:18.064867 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.064850    2560 scope.go:117] "RemoveContainer" containerID="ccbf4d8d18fd257618882abf6c85dedaa6916312d237f4f0e7c547888efe6c38"
Sep 09 03:30:18.065138 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:30:18.065121    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccbf4d8d18fd257618882abf6c85dedaa6916312d237f4f0e7c547888efe6c38\": container with ID starting with ccbf4d8d18fd257618882abf6c85dedaa6916312d237f4f0e7c547888efe6c38 not found: ID does not exist" containerID="ccbf4d8d18fd257618882abf6c85dedaa6916312d237f4f0e7c547888efe6c38"
Sep 09 03:30:18.065191 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.065146    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccbf4d8d18fd257618882abf6c85dedaa6916312d237f4f0e7c547888efe6c38"} err="failed to get container status \"ccbf4d8d18fd257618882abf6c85dedaa6916312d237f4f0e7c547888efe6c38\": rpc error: code = NotFound desc = could not find container \"ccbf4d8d18fd257618882abf6c85dedaa6916312d237f4f0e7c547888efe6c38\": container with ID starting with ccbf4d8d18fd257618882abf6c85dedaa6916312d237f4f0e7c547888efe6c38 not found: ID does not exist"
Sep 09 03:30:18.065191 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.065162    2560 scope.go:117] "RemoveContainer" containerID="6ee71488fcb02ecdc4f17aeb8a5209d2486d3ba72fdf7886c2a737bd974236ea"
Sep 09 03:30:18.065422 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:30:18.065403    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee71488fcb02ecdc4f17aeb8a5209d2486d3ba72fdf7886c2a737bd974236ea\": container with ID starting with 6ee71488fcb02ecdc4f17aeb8a5209d2486d3ba72fdf7886c2a737bd974236ea not found: ID does not exist" containerID="6ee71488fcb02ecdc4f17aeb8a5209d2486d3ba72fdf7886c2a737bd974236ea"
Sep 09 03:30:18.065470 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.065429    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee71488fcb02ecdc4f17aeb8a5209d2486d3ba72fdf7886c2a737bd974236ea"} err="failed to get container status \"6ee71488fcb02ecdc4f17aeb8a5209d2486d3ba72fdf7886c2a737bd974236ea\": rpc error: code = NotFound desc = could not find container \"6ee71488fcb02ecdc4f17aeb8a5209d2486d3ba72fdf7886c2a737bd974236ea\": container with ID starting with 6ee71488fcb02ecdc4f17aeb8a5209d2486d3ba72fdf7886c2a737bd974236ea not found: ID does not exist"
Sep 09 03:30:18.089303 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.089274    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"]
Sep 09 03:30:18.091127 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.091107    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9c982818-openshift-default-7dfd5bd896-rmpfq"]
Sep 09 03:30:18.130674 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.130646    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-envoy\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:18.130674 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.130675    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c65e186-03b1-4ad0-80d6-e2eacf40e324-credential-socket\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:18.130880 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.130690    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-podinfo\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:18.130880 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.130705    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v724\" (UniqueName: \"kubernetes.io/projected/9c65e186-03b1-4ad0-80d6-e2eacf40e324-kube-api-access-5v724\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:18.130880 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.130717    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c65e186-03b1-4ad0-80d6-e2eacf40e324-istio-token\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:18.135813 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.135782    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:18.352874 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:18.352843    2560 scope.go:117] "RemoveContainer" containerID="89b56ab89d4fe362044a153c7abce2279cfb3d10ad4ff71c0a0d6d67e71f05d0"
Sep 09 03:30:19.043341 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:19.043308    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8" event={"ID":"4fada035-7396-4acc-94fd-fab40bfaee87","Type":"ContainerStarted","Data":"7518f4ec520afdc3738456b94633688e203b264fdb878c0700cfe8aa88d78ba4"}
Sep 09 03:30:19.043565 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:19.043486    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:19.046005 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:19.045984    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2" event={"ID":"8c9f45fa-75fb-4fd6-b68a-138e05c6f672","Type":"ContainerStarted","Data":"f09d76b7712f3b846de938f0340c283c6a6e74c02dccca06c03f028e3b999a79"}
Sep 09 03:30:19.048198 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:19.048174    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:19.071863 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:19.071823    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8" podStartSLOduration=2.7584287610000002 podStartE2EDuration="3.071809599s" totalImagesPullingTime="313.380838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:30:19.069882125 +0000 UTC m=+1036.206629990" watchObservedRunningTime="2026-09-09 03:30:19.071809599 +0000 UTC m=+1036.208557464"
Sep 09 03:30:19.358776 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:19.358737    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c65e186-03b1-4ad0-80d6-e2eacf40e324" path="/var/lib/kubelet/pods/9c65e186-03b1-4ad0-80d6-e2eacf40e324/volumes"
Sep 09 03:30:19.969415 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:19.969381    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:20.053388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:20.053350    2560 generic.go:356] "Generic (PLEG): container finished" podID="8c9f45fa-75fb-4fd6-b68a-138e05c6f672" containerID="f09d76b7712f3b846de938f0340c283c6a6e74c02dccca06c03f028e3b999a79" exitCode=0
Sep 09 03:30:20.053550 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:20.053421    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2" event={"ID":"8c9f45fa-75fb-4fd6-b68a-138e05c6f672","Type":"ContainerDied","Data":"f09d76b7712f3b846de938f0340c283c6a6e74c02dccca06c03f028e3b999a79"}
Sep 09 03:30:20.053550 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:20.053479    2560 scope.go:117] "RemoveContainer" containerID="89b56ab89d4fe362044a153c7abce2279cfb3d10ad4ff71c0a0d6d67e71f05d0"
Sep 09 03:30:20.108533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:20.105494    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2"]
Sep 09 03:30:21.191728 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:21.191704    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2"
Sep 09 03:30:22.067060 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:22.067026    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2" event={"ID":"8c9f45fa-75fb-4fd6-b68a-138e05c6f672","Type":"ContainerDied","Data":"650a6d020cc26ef92d6c9ceaf0fdc23d308190ecc43224dee3184e7b7c4ce7fb"}
Sep 09 03:30:22.067060 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:22.067046    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2"
Sep 09 03:30:22.067270 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:22.067076    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="650a6d020cc26ef92d6c9ceaf0fdc23d308190ecc43224dee3184e7b7c4ce7fb"
Sep 09 03:30:22.097267 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:22.097237    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2"]
Sep 09 03:30:22.099009 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:22.098989    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4bd6b8-29815410-nf9k2"]
Sep 09 03:30:23.358317 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:23.358284    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c9f45fa-75fb-4fd6-b68a-138e05c6f672" path="/var/lib/kubelet/pods/8c9f45fa-75fb-4fd6-b68a-138e05c6f672/volumes"
Sep 09 03:30:23.572468 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:23.572433    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:24.905543 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:24.905511    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:30:25.058664 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:25.058637    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:36.144361 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:36.144326    2560 generic.go:356] "Generic (PLEG): container finished" podID="a3f31d25-6baa-48ce-ae38-4f8709bc0687" containerID="fad9557d38644b25b1434514cfc2e6afeb0024b9c0c0b4bc9872b90ce6b68566" exitCode=0
Sep 09 03:30:36.144898 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:36.144404    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg" event={"ID":"a3f31d25-6baa-48ce-ae38-4f8709bc0687","Type":"ContainerDied","Data":"fad9557d38644b25b1434514cfc2e6afeb0024b9c0c0b4bc9872b90ce6b68566"}
Sep 09 03:30:36.144964 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:36.144921    2560 scope.go:117] "RemoveContainer" containerID="fad9557d38644b25b1434514cfc2e6afeb0024b9c0c0b4bc9872b90ce6b68566"
Sep 09 03:30:36.146109 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:36.146081    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:30:36.146218 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:36.146149    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:30:36.146218 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:36.146180    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:30:37.151986 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:37.151948    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg" event={"ID":"a3f31d25-6baa-48ce-ae38-4f8709bc0687","Type":"ContainerStarted","Data":"6664d123b9b2b0b8f45b0aba749612e1b20df7a858f6e1cd2089532b4ba33433"}
Sep 09 03:30:38.046883 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:38.046850    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:38.047049 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:38.046991    2560 patch_prober.go:28] interesting pod/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 09 03:30:38.047100 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:38.047047    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg" podUID="a3f31d25-6baa-48ce-ae38-4f8709bc0687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 09 03:30:38.120343 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:38.120305    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:39.047309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:39.047269    2560 patch_prober.go:28] interesting pod/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 09 03:30:39.047716 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:39.047331    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg" podUID="a3f31d25-6baa-48ce-ae38-4f8709bc0687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 09 03:30:40.047130 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:40.047093    2560 patch_prober.go:28] interesting pod/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 09 03:30:40.047323 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:40.047162    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg" podUID="a3f31d25-6baa-48ce-ae38-4f8709bc0687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 09 03:30:41.047179 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:41.047142    2560 patch_prober.go:28] interesting pod/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 09 03:30:41.047584 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:41.047201    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg" podUID="a3f31d25-6baa-48ce-ae38-4f8709bc0687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 09 03:30:42.047590 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:42.047553    2560 patch_prober.go:28] interesting pod/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 09 03:30:42.047976 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:42.047640    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg" podUID="a3f31d25-6baa-48ce-ae38-4f8709bc0687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 09 03:30:43.051062 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:43.051030    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:43.051426 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:43.051345    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:43.052021 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:43.052004    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:46.396690 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:46.396660    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:30:48.056619 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.056563    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"]
Sep 09 03:30:48.057063 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.056808    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8" podUID="4fada035-7396-4acc-94fd-fab40bfaee87" containerName="maas-api" containerID="cri-o://7518f4ec520afdc3738456b94633688e203b264fdb878c0700cfe8aa88d78ba4" gracePeriod=30
Sep 09 03:30:48.094356 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.094322    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:48.163703 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.163662    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:48.233490 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.233458    2560 generic.go:356] "Generic (PLEG): container finished" podID="4fada035-7396-4acc-94fd-fab40bfaee87" containerID="7518f4ec520afdc3738456b94633688e203b264fdb878c0700cfe8aa88d78ba4" exitCode=0
Sep 09 03:30:48.233638 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.233494    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8" event={"ID":"4fada035-7396-4acc-94fd-fab40bfaee87","Type":"ContainerDied","Data":"7518f4ec520afdc3738456b94633688e203b264fdb878c0700cfe8aa88d78ba4"}
Sep 09 03:30:48.260105 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.260058    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:48.307815 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.307747    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:48.349772 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.349737    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:48.443650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.443490    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fada035-7396-4acc-94fd-fab40bfaee87-maas-api-tls\") pod \"4fada035-7396-4acc-94fd-fab40bfaee87\" (UID: \"4fada035-7396-4acc-94fd-fab40bfaee87\") "
Sep 09 03:30:48.443827 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.443751    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfkln\" (UniqueName: \"kubernetes.io/projected/4fada035-7396-4acc-94fd-fab40bfaee87-kube-api-access-zfkln\") pod \"4fada035-7396-4acc-94fd-fab40bfaee87\" (UID: \"4fada035-7396-4acc-94fd-fab40bfaee87\") "
Sep 09 03:30:48.443827 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.443789    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fada035-7396-4acc-94fd-fab40bfaee87-metrics-certs\") pod \"4fada035-7396-4acc-94fd-fab40bfaee87\" (UID: \"4fada035-7396-4acc-94fd-fab40bfaee87\") "
Sep 09 03:30:48.443936 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.443875    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fada035-7396-4acc-94fd-fab40bfaee87-openshift-service-ca\") pod \"4fada035-7396-4acc-94fd-fab40bfaee87\" (UID: \"4fada035-7396-4acc-94fd-fab40bfaee87\") "
Sep 09 03:30:48.444584 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.444552    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fada035-7396-4acc-94fd-fab40bfaee87-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4fada035-7396-4acc-94fd-fab40bfaee87" (UID: "4fada035-7396-4acc-94fd-fab40bfaee87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:30:48.446241 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.446208    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fada035-7396-4acc-94fd-fab40bfaee87-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4fada035-7396-4acc-94fd-fab40bfaee87" (UID: "4fada035-7396-4acc-94fd-fab40bfaee87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:30:48.446497 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.446480    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fada035-7396-4acc-94fd-fab40bfaee87-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4fada035-7396-4acc-94fd-fab40bfaee87" (UID: "4fada035-7396-4acc-94fd-fab40bfaee87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:30:48.446584 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.446569    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fada035-7396-4acc-94fd-fab40bfaee87-kube-api-access-zfkln" (OuterVolumeSpecName: "kube-api-access-zfkln") pod "4fada035-7396-4acc-94fd-fab40bfaee87" (UID: "4fada035-7396-4acc-94fd-fab40bfaee87"). InnerVolumeSpecName "kube-api-access-zfkln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:30:48.480515 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.480478    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"]
Sep 09 03:30:48.480769 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.480740    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg" podUID="a3f31d25-6baa-48ce-ae38-4f8709bc0687" containerName="istio-proxy" containerID="cri-o://6664d123b9b2b0b8f45b0aba749612e1b20df7a858f6e1cd2089532b4ba33433" gracePeriod=30
Sep 09 03:30:48.503369 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.503336    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:48.545051 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.545023    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fada035-7396-4acc-94fd-fab40bfaee87-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:48.545051 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.545046    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfkln\" (UniqueName: \"kubernetes.io/projected/4fada035-7396-4acc-94fd-fab40bfaee87-kube-api-access-zfkln\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:48.545051 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.545055    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fada035-7396-4acc-94fd-fab40bfaee87-metrics-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:48.545263 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.545064    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fada035-7396-4acc-94fd-fab40bfaee87-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:48.972925 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:48.972881    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:49.239820 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.239743    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"
Sep 09 03:30:49.240233 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.239739    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8" event={"ID":"4fada035-7396-4acc-94fd-fab40bfaee87","Type":"ContainerDied","Data":"878681d3102c0f7bed2f05198eaad296536f9577a83b1c108ebb232f30e4be94"}
Sep 09 03:30:49.240233 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.239886    2560 scope.go:117] "RemoveContainer" containerID="7518f4ec520afdc3738456b94633688e203b264fdb878c0700cfe8aa88d78ba4"
Sep 09 03:30:49.273159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.273124    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"]
Sep 09 03:30:49.286069 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.286038    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f9aae59-7f4d9c566c-ftkj8"]
Sep 09 03:30:49.287989 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.287967    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"]
Sep 09 03:30:49.288384 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.288369    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c9f45fa-75fb-4fd6-b68a-138e05c6f672" containerName="cleanup"
Sep 09 03:30:49.288425 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.288387    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c9f45fa-75fb-4fd6-b68a-138e05c6f672" containerName="cleanup"
Sep 09 03:30:49.288425 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.288406    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fada035-7396-4acc-94fd-fab40bfaee87" containerName="maas-api"
Sep 09 03:30:49.288425 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.288411    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fada035-7396-4acc-94fd-fab40bfaee87" containerName="maas-api"
Sep 09 03:30:49.288425 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.288420    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c65e186-03b1-4ad0-80d6-e2eacf40e324" containerName="istio-proxy"
Sep 09 03:30:49.288425 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.288426    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c65e186-03b1-4ad0-80d6-e2eacf40e324" containerName="istio-proxy"
Sep 09 03:30:49.288579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.288436    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c9f45fa-75fb-4fd6-b68a-138e05c6f672" containerName="cleanup"
Sep 09 03:30:49.288579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.288441    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c9f45fa-75fb-4fd6-b68a-138e05c6f672" containerName="cleanup"
Sep 09 03:30:49.288579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.288519    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c9f45fa-75fb-4fd6-b68a-138e05c6f672" containerName="cleanup"
Sep 09 03:30:49.288579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.288528    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c65e186-03b1-4ad0-80d6-e2eacf40e324" containerName="istio-proxy"
Sep 09 03:30:49.288579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.288537    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fada035-7396-4acc-94fd-fab40bfaee87" containerName="maas-api"
Sep 09 03:30:49.288579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.288546    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c9f45fa-75fb-4fd6-b68a-138e05c6f672" containerName="cleanup"
Sep 09 03:30:49.288579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.288553    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c65e186-03b1-4ad0-80d6-e2eacf40e324" containerName="istio-proxy"
Sep 09 03:30:49.293007 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.292989    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.296091 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.296067    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b0f5d040-openshift-default-dockercfg-55dlp\""
Sep 09 03:30:49.309800 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.309777    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"]
Sep 09 03:30:49.327169 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.327140    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:49.359397 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.359360    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fada035-7396-4acc-94fd-fab40bfaee87" path="/var/lib/kubelet/pods/4fada035-7396-4acc-94fd-fab40bfaee87/volumes"
Sep 09 03:30:49.452733 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.452698    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-data\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.452910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.452738    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-podinfo\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.452910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.452801    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istiod-ca-cert\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.452910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.452818    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blnsq\" (UniqueName: \"kubernetes.io/projected/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-kube-api-access-blnsq\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.452910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.452838    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-credential-socket\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.452910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.452888    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-token\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.453179 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.452961    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-workload-certs\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.453179 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.453049    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-workload-socket\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.453179 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.453085    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-envoy\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.554542 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.554445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-workload-certs\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.554542 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.554505    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-workload-socket\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.554798 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.554643    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-envoy\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.554798 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.554715    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-data\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.554798 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.554744    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-podinfo\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.554798 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.554783    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istiod-ca-cert\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.555089 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.554804    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blnsq\" (UniqueName: \"kubernetes.io/projected/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-kube-api-access-blnsq\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.555089 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.554831    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-credential-socket\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.555089 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.554854    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-token\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.555089 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.554906    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-workload-socket\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.555089 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.554959    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-workload-certs\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.555325 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.555112    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-data\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.555522 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.555499    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-credential-socket\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.555678 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.555655    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istiod-ca-cert\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.556980 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.556958    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-envoy\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.557326 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.557307    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-podinfo\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.562893 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.562865    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-token\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.563014 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.562997    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blnsq\" (UniqueName: \"kubernetes.io/projected/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-kube-api-access-blnsq\") pod \"e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") " pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.604033 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.603992    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:49.638686 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.638648    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:49.881295 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.881271    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"]
Sep 09 03:30:49.883401 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:30:49.883371    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8a1cf2_1fcf_4910_9e69_de13e7342b4d.slice/crio-662fdf67db7816b79e9e551ba2efcf10385f5cf7d1d55f73271f5d1c36a2adff WatchSource:0}: Error finding container 662fdf67db7816b79e9e551ba2efcf10385f5cf7d1d55f73271f5d1c36a2adff: Status 404 returned error can't find the container with id 662fdf67db7816b79e9e551ba2efcf10385f5cf7d1d55f73271f5d1c36a2adff
Sep 09 03:30:49.885809 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.885771    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:30:49.885881 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.885858    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:30:49.885920 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:49.885900    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:30:50.247584 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:50.247489    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4" event={"ID":"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d","Type":"ContainerStarted","Data":"b241a72486fbb6ff364bc4a5e85299aa48897305241dcd51a8d2a0cea506c237"}
Sep 09 03:30:50.247584 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:50.247546    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4" event={"ID":"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d","Type":"ContainerStarted","Data":"662fdf67db7816b79e9e551ba2efcf10385f5cf7d1d55f73271f5d1c36a2adff"}
Sep 09 03:30:50.270279 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:50.270226    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4" podStartSLOduration=1.270211149 podStartE2EDuration="1.270211149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:30:50.268696441 +0000 UTC m=+1067.405444306" watchObservedRunningTime="2026-09-09 03:30:50.270211149 +0000 UTC m=+1067.406959013"
Sep 09 03:30:50.604413 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:50.604380    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:50.609702 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:50.609677    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:51.252950 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:51.252920    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:51.253847 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:51.253827    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:30:51.621199 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:51.621150    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:51.709073 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:51.709034    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:52.735715 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.735679    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"]
Sep 09 03:30:52.739084 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.737063    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c9f45fa-75fb-4fd6-b68a-138e05c6f672" containerName="cleanup"
Sep 09 03:30:52.739084 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.737087    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c9f45fa-75fb-4fd6-b68a-138e05c6f672" containerName="cleanup"
Sep 09 03:30:52.739084 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.737145    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c65e186-03b1-4ad0-80d6-e2eacf40e324" containerName="istio-proxy"
Sep 09 03:30:52.739084 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.737154    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c65e186-03b1-4ad0-80d6-e2eacf40e324" containerName="istio-proxy"
Sep 09 03:30:52.739084 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.737375    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c9f45fa-75fb-4fd6-b68a-138e05c6f672" containerName="cleanup"
Sep 09 03:30:52.742486 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.742458    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"
Sep 09 03:30:52.746164 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.746131    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b0f5d040-serving-cert\""
Sep 09 03:30:52.751684 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.751481    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"]
Sep 09 03:30:52.771198 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.771162    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:52.874984 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.874947    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:52.888735 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.888700    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpdn7\" (UniqueName: \"kubernetes.io/projected/26208f8a-7662-410e-b91c-96797f43fe35-kube-api-access-fpdn7\") pod \"maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq\" (UID: \"26208f8a-7662-410e-b91c-96797f43fe35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"
Sep 09 03:30:52.888735 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.888738    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26208f8a-7662-410e-b91c-96797f43fe35-metrics-certs\") pod \"maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq\" (UID: \"26208f8a-7662-410e-b91c-96797f43fe35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"
Sep 09 03:30:52.888905 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.888782    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26208f8a-7662-410e-b91c-96797f43fe35-maas-api-tls\") pod \"maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq\" (UID: \"26208f8a-7662-410e-b91c-96797f43fe35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"
Sep 09 03:30:52.888905 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.888816    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26208f8a-7662-410e-b91c-96797f43fe35-openshift-service-ca\") pod \"maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq\" (UID: \"26208f8a-7662-410e-b91c-96797f43fe35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"
Sep 09 03:30:52.967667 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.967633    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:52.989883 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.989811    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpdn7\" (UniqueName: \"kubernetes.io/projected/26208f8a-7662-410e-b91c-96797f43fe35-kube-api-access-fpdn7\") pod \"maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq\" (UID: \"26208f8a-7662-410e-b91c-96797f43fe35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"
Sep 09 03:30:52.989883 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.989850    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26208f8a-7662-410e-b91c-96797f43fe35-metrics-certs\") pod \"maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq\" (UID: \"26208f8a-7662-410e-b91c-96797f43fe35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"
Sep 09 03:30:52.990043 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.989890    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26208f8a-7662-410e-b91c-96797f43fe35-maas-api-tls\") pod \"maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq\" (UID: \"26208f8a-7662-410e-b91c-96797f43fe35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"
Sep 09 03:30:52.990043 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.989913    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26208f8a-7662-410e-b91c-96797f43fe35-openshift-service-ca\") pod \"maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq\" (UID: \"26208f8a-7662-410e-b91c-96797f43fe35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"
Sep 09 03:30:52.990556 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.990533    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26208f8a-7662-410e-b91c-96797f43fe35-openshift-service-ca\") pod \"maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq\" (UID: \"26208f8a-7662-410e-b91c-96797f43fe35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"
Sep 09 03:30:52.992307 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.992282    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26208f8a-7662-410e-b91c-96797f43fe35-metrics-certs\") pod \"maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq\" (UID: \"26208f8a-7662-410e-b91c-96797f43fe35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"
Sep 09 03:30:52.992412 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.992393    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26208f8a-7662-410e-b91c-96797f43fe35-maas-api-tls\") pod \"maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq\" (UID: \"26208f8a-7662-410e-b91c-96797f43fe35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"
Sep 09 03:30:52.998721 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:52.998697    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpdn7\" (UniqueName: \"kubernetes.io/projected/26208f8a-7662-410e-b91c-96797f43fe35-kube-api-access-fpdn7\") pod \"maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq\" (UID: \"26208f8a-7662-410e-b91c-96797f43fe35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"
Sep 09 03:30:53.059530 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.059494    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"
Sep 09 03:30:53.301110 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.300988    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:53.336891 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.336866    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"]
Sep 09 03:30:53.338689 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:30:53.338661    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26208f8a_7662_410e_b91c_96797f43fe35.slice/crio-360ed115b2c2634880a3c5928ec924a3d42992be601bc709ddf8f69cd41748b5 WatchSource:0}: Error finding container 360ed115b2c2634880a3c5928ec924a3d42992be601bc709ddf8f69cd41748b5: Status 404 returned error can't find the container with id 360ed115b2c2634880a3c5928ec924a3d42992be601bc709ddf8f69cd41748b5
Sep 09 03:30:53.396767 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.396729    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:53.727692 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.727667    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:53.736484 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.736445    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:53.896954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.896930    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-token\") pod \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") "
Sep 09 03:30:53.897048 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.896976    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-workload-socket\") pod \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") "
Sep 09 03:30:53.897048 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.897021    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istiod-ca-cert\") pod \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") "
Sep 09 03:30:53.897119 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.897047    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-workload-certs\") pod \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") "
Sep 09 03:30:53.897119 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.897084    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-data\") pod \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") "
Sep 09 03:30:53.897190 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.897137    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-credential-socket\") pod \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") "
Sep 09 03:30:53.897190 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.897168    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwfck\" (UniqueName: \"kubernetes.io/projected/a3f31d25-6baa-48ce-ae38-4f8709bc0687-kube-api-access-xwfck\") pod \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") "
Sep 09 03:30:53.897723 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.897364    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a3f31d25-6baa-48ce-ae38-4f8709bc0687" (UID: "a3f31d25-6baa-48ce-ae38-4f8709bc0687"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:30:53.897723 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.897417    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a3f31d25-6baa-48ce-ae38-4f8709bc0687" (UID: "a3f31d25-6baa-48ce-ae38-4f8709bc0687"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:30:53.897723 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.897498    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-podinfo\") pod \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") "
Sep 09 03:30:53.897723 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.897541    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-data" (OuterVolumeSpecName: "istio-data") pod "a3f31d25-6baa-48ce-ae38-4f8709bc0687" (UID: "a3f31d25-6baa-48ce-ae38-4f8709bc0687"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:30:53.897723 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.897569    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-envoy\") pod \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\" (UID: \"a3f31d25-6baa-48ce-ae38-4f8709bc0687\") "
Sep 09 03:30:53.897723 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.897608    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a3f31d25-6baa-48ce-ae38-4f8709bc0687" (UID: "a3f31d25-6baa-48ce-ae38-4f8709bc0687"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:30:53.897723 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.897631    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a3f31d25-6baa-48ce-ae38-4f8709bc0687" (UID: "a3f31d25-6baa-48ce-ae38-4f8709bc0687"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:30:53.898133 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.897963    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-workload-socket\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:53.898133 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.897985    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istiod-ca-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:53.898133 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.898001    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-workload-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:53.898133 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.898013    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-data\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:53.898133 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.898026    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-credential-socket\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:53.899583 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.899557    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-token" (OuterVolumeSpecName: "istio-token") pod "a3f31d25-6baa-48ce-ae38-4f8709bc0687" (UID: "a3f31d25-6baa-48ce-ae38-4f8709bc0687"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:30:53.899978 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.899952    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a3f31d25-6baa-48ce-ae38-4f8709bc0687" (UID: "a3f31d25-6baa-48ce-ae38-4f8709bc0687"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:30:53.900068 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.899981    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f31d25-6baa-48ce-ae38-4f8709bc0687-kube-api-access-xwfck" (OuterVolumeSpecName: "kube-api-access-xwfck") pod "a3f31d25-6baa-48ce-ae38-4f8709bc0687" (UID: "a3f31d25-6baa-48ce-ae38-4f8709bc0687"). InnerVolumeSpecName "kube-api-access-xwfck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:30:53.900068 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.900010    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a3f31d25-6baa-48ce-ae38-4f8709bc0687" (UID: "a3f31d25-6baa-48ce-ae38-4f8709bc0687"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:30:53.998860 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.998825    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-token\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:53.998860 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.998858    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwfck\" (UniqueName: \"kubernetes.io/projected/a3f31d25-6baa-48ce-ae38-4f8709bc0687-kube-api-access-xwfck\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:53.999070 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.998870    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-podinfo\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:53.999070 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:53.998878    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f31d25-6baa-48ce-ae38-4f8709bc0687-istio-envoy\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:30:54.270630 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:54.270523    2560 generic.go:356] "Generic (PLEG): container finished" podID="a3f31d25-6baa-48ce-ae38-4f8709bc0687" containerID="6664d123b9b2b0b8f45b0aba749612e1b20df7a858f6e1cd2089532b4ba33433" exitCode=0
Sep 09 03:30:54.270630 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:54.270608    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"
Sep 09 03:30:54.270871 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:54.270640    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg" event={"ID":"a3f31d25-6baa-48ce-ae38-4f8709bc0687","Type":"ContainerDied","Data":"6664d123b9b2b0b8f45b0aba749612e1b20df7a858f6e1cd2089532b4ba33433"}
Sep 09 03:30:54.270871 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:54.270716    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg" event={"ID":"a3f31d25-6baa-48ce-ae38-4f8709bc0687","Type":"ContainerDied","Data":"3c0d4434b9a2fd81967b61a4949979b99bb76ebf79b14f9b59ba795506e9ba29"}
Sep 09 03:30:54.270871 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:54.270778    2560 scope.go:117] "RemoveContainer" containerID="6664d123b9b2b0b8f45b0aba749612e1b20df7a858f6e1cd2089532b4ba33433"
Sep 09 03:30:54.272544 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:54.272517    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq" event={"ID":"26208f8a-7662-410e-b91c-96797f43fe35","Type":"ContainerStarted","Data":"d21fffb154462b0b3b1fd387edd5ac67ce9e6055c08163fab6734a42621e7dca"}
Sep 09 03:30:54.272696 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:54.272566    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq" event={"ID":"26208f8a-7662-410e-b91c-96797f43fe35","Type":"ContainerStarted","Data":"360ed115b2c2634880a3c5928ec924a3d42992be601bc709ddf8f69cd41748b5"}
Sep 09 03:30:54.280529 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:54.280512    2560 scope.go:117] "RemoveContainer" containerID="fad9557d38644b25b1434514cfc2e6afeb0024b9c0c0b4bc9872b90ce6b68566"
Sep 09 03:30:54.288517 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:54.288499    2560 scope.go:117] "RemoveContainer" containerID="6664d123b9b2b0b8f45b0aba749612e1b20df7a858f6e1cd2089532b4ba33433"
Sep 09 03:30:54.288799 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:30:54.288782    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6664d123b9b2b0b8f45b0aba749612e1b20df7a858f6e1cd2089532b4ba33433\": container with ID starting with 6664d123b9b2b0b8f45b0aba749612e1b20df7a858f6e1cd2089532b4ba33433 not found: ID does not exist" containerID="6664d123b9b2b0b8f45b0aba749612e1b20df7a858f6e1cd2089532b4ba33433"
Sep 09 03:30:54.288849 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:54.288809    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6664d123b9b2b0b8f45b0aba749612e1b20df7a858f6e1cd2089532b4ba33433"} err="failed to get container status \"6664d123b9b2b0b8f45b0aba749612e1b20df7a858f6e1cd2089532b4ba33433\": rpc error: code = NotFound desc = could not find container \"6664d123b9b2b0b8f45b0aba749612e1b20df7a858f6e1cd2089532b4ba33433\": container with ID starting with 6664d123b9b2b0b8f45b0aba749612e1b20df7a858f6e1cd2089532b4ba33433 not found: ID does not exist"
Sep 09 03:30:54.288849 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:54.288824    2560 scope.go:117] "RemoveContainer" containerID="fad9557d38644b25b1434514cfc2e6afeb0024b9c0c0b4bc9872b90ce6b68566"
Sep 09 03:30:54.289076 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:30:54.289061    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fad9557d38644b25b1434514cfc2e6afeb0024b9c0c0b4bc9872b90ce6b68566\": container with ID starting with fad9557d38644b25b1434514cfc2e6afeb0024b9c0c0b4bc9872b90ce6b68566 not found: ID does not exist" containerID="fad9557d38644b25b1434514cfc2e6afeb0024b9c0c0b4bc9872b90ce6b68566"
Sep 09 03:30:54.289114 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:54.289083    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad9557d38644b25b1434514cfc2e6afeb0024b9c0c0b4bc9872b90ce6b68566"} err="failed to get container status \"fad9557d38644b25b1434514cfc2e6afeb0024b9c0c0b4bc9872b90ce6b68566\": rpc error: code = NotFound desc = could not find container \"fad9557d38644b25b1434514cfc2e6afeb0024b9c0c0b4bc9872b90ce6b68566\": container with ID starting with fad9557d38644b25b1434514cfc2e6afeb0024b9c0c0b4bc9872b90ce6b68566 not found: ID does not exist"
Sep 09 03:30:54.299546 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:54.299506    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq" podStartSLOduration=1.780410146 podStartE2EDuration="2.299492522s" totalImagesPullingTime="519.082376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:30:54.29834975 +0000 UTC m=+1071.435097640" watchObservedRunningTime="2026-09-09 03:30:54.299492522 +0000 UTC m=+1071.436240386"
Sep 09 03:30:54.318442 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:54.318414    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"]
Sep 09 03:30:54.322959 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:54.322936    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-5f9aae59-gw-openshift-default-5866f8768-vgkmg"]
Sep 09 03:30:55.279221 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:55.279195    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"
Sep 09 03:30:55.358565 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:55.358535    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3f31d25-6baa-48ce-ae38-4f8709bc0687" path="/var/lib/kubelet/pods/a3f31d25-6baa-48ce-ae38-4f8709bc0687/volumes"
Sep 09 03:30:59.178027 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.177955    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"]
Sep 09 03:30:59.178411 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.178395    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3f31d25-6baa-48ce-ae38-4f8709bc0687" containerName="istio-proxy"
Sep 09 03:30:59.178411 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.178412    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3f31d25-6baa-48ce-ae38-4f8709bc0687" containerName="istio-proxy"
Sep 09 03:30:59.178526 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.178420    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3f31d25-6baa-48ce-ae38-4f8709bc0687" containerName="istio-proxy"
Sep 09 03:30:59.178526 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.178425    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3f31d25-6baa-48ce-ae38-4f8709bc0687" containerName="istio-proxy"
Sep 09 03:30:59.178526 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.178498    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3f31d25-6baa-48ce-ae38-4f8709bc0687" containerName="istio-proxy"
Sep 09 03:30:59.178526 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.178506    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3f31d25-6baa-48ce-ae38-4f8709bc0687" containerName="istio-proxy"
Sep 09 03:30:59.183185 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.183166    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.185891 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.185869    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-e893b71c-kserve-self-signed-certs\""
Sep 09 03:30:59.191032 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.191011    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"]
Sep 09 03:30:59.247235 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.247208    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-model-cache\") pod \"e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") " pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.247376 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.247245    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm5nw\" (UniqueName: \"kubernetes.io/projected/abe67204-2ca8-499c-be1d-53fa2bc3e345-kube-api-access-mm5nw\") pod \"e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") " pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.247376 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.247266    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-dshm\") pod \"e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") " pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.247376 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.247359    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe67204-2ca8-499c-be1d-53fa2bc3e345-tls-certs\") pod \"e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") " pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.247496 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.247451    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-tmp-dir\") pod \"e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") " pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.247496 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.247470    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-home\") pod \"e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") " pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.348925 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.348882    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-tmp-dir\") pod \"e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") " pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.348925 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.348926    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-home\") pod \"e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") " pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.349162 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.348965    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-model-cache\") pod \"e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") " pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.349162 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.348987    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm5nw\" (UniqueName: \"kubernetes.io/projected/abe67204-2ca8-499c-be1d-53fa2bc3e345-kube-api-access-mm5nw\") pod \"e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") " pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.349162 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.349004    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-dshm\") pod \"e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") " pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.349162 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.349035    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe67204-2ca8-499c-be1d-53fa2bc3e345-tls-certs\") pod \"e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") " pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.349399 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.349286    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-tmp-dir\") pod \"e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") " pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.349399 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.349336    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-model-cache\") pod \"e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") " pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.349399 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.349376    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-home\") pod \"e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") " pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.351339 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.351319    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-dshm\") pod \"e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") " pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.351627 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.351608    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe67204-2ca8-499c-be1d-53fa2bc3e345-tls-certs\") pod \"e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") " pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.358519 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.358498    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm5nw\" (UniqueName: \"kubernetes.io/projected/abe67204-2ca8-499c-be1d-53fa2bc3e345-kube-api-access-mm5nw\") pod \"e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") " pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.493020 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.492929    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:30:59.575346 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.575314    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:30:59.768811 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:30:59.768784    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"]
Sep 09 03:30:59.771056 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:30:59.771027    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabe67204_2ca8_499c_be1d_53fa2bc3e345.slice/crio-cd3592fffda366410ccc5f3ea1790cfd70155c23f552e41f3b8a2bea5db39bcc WatchSource:0}: Error finding container cd3592fffda366410ccc5f3ea1790cfd70155c23f552e41f3b8a2bea5db39bcc: Status 404 returned error can't find the container with id cd3592fffda366410ccc5f3ea1790cfd70155c23f552e41f3b8a2bea5db39bcc
Sep 09 03:31:00.306923 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:00.306885    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm" event={"ID":"abe67204-2ca8-499c-be1d-53fa2bc3e345","Type":"ContainerStarted","Data":"7b5b01077db87a5e849cf207c4c1761cc5678e878e27598b97e06e4a9acaad94"}
Sep 09 03:31:00.307338 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:00.306932    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm" event={"ID":"abe67204-2ca8-499c-be1d-53fa2bc3e345","Type":"ContainerStarted","Data":"cd3592fffda366410ccc5f3ea1790cfd70155c23f552e41f3b8a2bea5db39bcc"}
Sep 09 03:31:00.325853 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:00.325801    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm" podStartSLOduration=1.325750662 podStartE2EDuration="1.325750662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:31:00.324187387 +0000 UTC m=+1077.460935252" watchObservedRunningTime="2026-09-09 03:31:00.325750662 +0000 UTC m=+1077.462498535"
Sep 09 03:31:01.290232 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:01.290200    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"
Sep 09 03:31:01.802830 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:01.802795    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:31:09.493260 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:09.493210    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:31:09.493260 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:09.493270    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:31:09.508250 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:09.508221    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:31:10.371158 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:10.371127    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:31:12.372154 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:12.372123    2560 generic.go:356] "Generic (PLEG): container finished" podID="1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" containerID="b241a72486fbb6ff364bc4a5e85299aa48897305241dcd51a8d2a0cea506c237" exitCode=0
Sep 09 03:31:12.372637 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:12.372188    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4" event={"ID":"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d","Type":"ContainerDied","Data":"b241a72486fbb6ff364bc4a5e85299aa48897305241dcd51a8d2a0cea506c237"}
Sep 09 03:31:12.372637 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:12.372575    2560 scope.go:117] "RemoveContainer" containerID="b241a72486fbb6ff364bc4a5e85299aa48897305241dcd51a8d2a0cea506c237"
Sep 09 03:31:12.373845 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:12.373807    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:31:12.373932 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:12.373877    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:31:12.373932 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:12.373908    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:31:13.392206 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.392169    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4" event={"ID":"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d","Type":"ContainerStarted","Data":"a105ccdea25eecda6b78f4ea39c114086a2fb3b8c63a639d7a182f1c37c21853"}
Sep 09 03:31:13.482443 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.482406    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"]
Sep 09 03:31:13.486922 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.486899    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.490990 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.490965    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a2fc47\"/\"openshift-service-ca.crt\""
Sep 09 03:31:13.491230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.490974    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a2fc47\"/\"default-dockercfg-xv8r2\""
Sep 09 03:31:13.491333 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.491028    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a2fc47\"/\"autores-model-a2fc47-kserve-self-signed-certs\""
Sep 09 03:31:13.491333 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.491040    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a2fc47\"/\"kube-root-ca.crt\""
Sep 09 03:31:13.497306 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.497284    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"]
Sep 09 03:31:13.519256 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.519228    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:13.587433 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.587397    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-model-cache\") pod \"autores-model-a2fc47-kserve-7b89b54b6c-wq5f4\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") " pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.587433 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.587434    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-dshm\") pod \"autores-model-a2fc47-kserve-7b89b54b6c-wq5f4\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") " pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.587674 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.587457    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb93bc05-0816-4395-a76c-ced478c9c49c-tls-certs\") pod \"autores-model-a2fc47-kserve-7b89b54b6c-wq5f4\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") " pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.587674 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.587531    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s65b4\" (UniqueName: \"kubernetes.io/projected/bb93bc05-0816-4395-a76c-ced478c9c49c-kube-api-access-s65b4\") pod \"autores-model-a2fc47-kserve-7b89b54b6c-wq5f4\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") " pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.587674 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.587565    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-home\") pod \"autores-model-a2fc47-kserve-7b89b54b6c-wq5f4\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") " pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.587775 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.587666    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-tmp-dir\") pod \"autores-model-a2fc47-kserve-7b89b54b6c-wq5f4\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") " pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.604525 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.604491    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:31:13.604815 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.604794    2560 patch_prober.go:28] interesting pod/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 09 03:31:13.604922 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.604858    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4" podUID="1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 09 03:31:13.689264 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.689177    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-dshm\") pod \"autores-model-a2fc47-kserve-7b89b54b6c-wq5f4\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") " pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.689264 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.689219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb93bc05-0816-4395-a76c-ced478c9c49c-tls-certs\") pod \"autores-model-a2fc47-kserve-7b89b54b6c-wq5f4\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") " pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.689264 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.689255    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s65b4\" (UniqueName: \"kubernetes.io/projected/bb93bc05-0816-4395-a76c-ced478c9c49c-kube-api-access-s65b4\") pod \"autores-model-a2fc47-kserve-7b89b54b6c-wq5f4\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") " pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.689513 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.689286    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-home\") pod \"autores-model-a2fc47-kserve-7b89b54b6c-wq5f4\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") " pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.689513 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.689327    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-tmp-dir\") pod \"autores-model-a2fc47-kserve-7b89b54b6c-wq5f4\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") " pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.689513 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.689377    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-model-cache\") pod \"autores-model-a2fc47-kserve-7b89b54b6c-wq5f4\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") " pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.689794 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.689761    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-home\") pod \"autores-model-a2fc47-kserve-7b89b54b6c-wq5f4\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") " pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.689934 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.689793    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-model-cache\") pod \"autores-model-a2fc47-kserve-7b89b54b6c-wq5f4\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") " pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.689934 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.689824    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-tmp-dir\") pod \"autores-model-a2fc47-kserve-7b89b54b6c-wq5f4\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") " pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.691650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.691630    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-dshm\") pod \"autores-model-a2fc47-kserve-7b89b54b6c-wq5f4\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") " pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.691746 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.691715    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb93bc05-0816-4395-a76c-ced478c9c49c-tls-certs\") pod \"autores-model-a2fc47-kserve-7b89b54b6c-wq5f4\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") " pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.699462 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.699439    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s65b4\" (UniqueName: \"kubernetes.io/projected/bb93bc05-0816-4395-a76c-ced478c9c49c-kube-api-access-s65b4\") pod \"autores-model-a2fc47-kserve-7b89b54b6c-wq5f4\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") " pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:13.798658 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:13.798586    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:14.085283 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:14.085104    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"]
Sep 09 03:31:14.087548 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:31:14.087520    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb93bc05_0816_4395_a76c_ced478c9c49c.slice/crio-88d88295a99ec4058b1a8779d5fd684156c95175a2c05b066d148bb97199b05b WatchSource:0}: Error finding container 88d88295a99ec4058b1a8779d5fd684156c95175a2c05b066d148bb97199b05b: Status 404 returned error can't find the container with id 88d88295a99ec4058b1a8779d5fd684156c95175a2c05b066d148bb97199b05b
Sep 09 03:31:14.398800 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:14.398758    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4" event={"ID":"bb93bc05-0816-4395-a76c-ced478c9c49c","Type":"ContainerStarted","Data":"b1011a469b692db33222d9ac44c4f6865e2cf896d52fe96bfac5491e99949a87"}
Sep 09 03:31:14.399191 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:14.398809    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4" event={"ID":"bb93bc05-0816-4395-a76c-ced478c9c49c","Type":"ContainerStarted","Data":"88d88295a99ec4058b1a8779d5fd684156c95175a2c05b066d148bb97199b05b"}
Sep 09 03:31:14.422745 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:14.422677    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4" podStartSLOduration=1.422661084 podStartE2EDuration="1.422661084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:31:14.421440398 +0000 UTC m=+1091.558188263" watchObservedRunningTime="2026-09-09 03:31:14.422661084 +0000 UTC m=+1091.559408951"
Sep 09 03:31:14.604922 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:14.604881    2560 patch_prober.go:28] interesting pod/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 09 03:31:14.605063 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:14.604951    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4" podUID="1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 09 03:31:15.604919 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:15.604883    2560 patch_prober.go:28] interesting pod/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 09 03:31:15.605301 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:15.604935    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4" podUID="1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 09 03:31:16.604814 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:16.604775    2560 patch_prober.go:28] interesting pod/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 09 03:31:16.604985 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:16.604835    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4" podUID="1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 09 03:31:16.758724 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:16.758691    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:31:16.759354 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:16.759250    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:17.605388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:17.605345    2560 patch_prober.go:28] interesting pod/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 09 03:31:17.605957 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:17.605416    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4" podUID="1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 09 03:31:17.833668 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:17.833634    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:18.608253 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:18.608220    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:31:18.608698 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:18.608505    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:31:18.609269 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:18.609254    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:31:23.474697 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.474662    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"]
Sep 09 03:31:23.475069 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.474863    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq" podUID="26208f8a-7662-410e-b91c-96797f43fe35" containerName="maas-api" containerID="cri-o://d21fffb154462b0b3b1fd387edd5ac67ce9e6055c08163fab6734a42621e7dca" gracePeriod=30
Sep 09 03:31:23.506059 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.506011    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:23.598748 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.598719    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:23.669975 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.669937    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:23.722762 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.722744    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"
Sep 09 03:31:23.736878 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.736813    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:23.789728 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.789701    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26208f8a-7662-410e-b91c-96797f43fe35-metrics-certs\") pod \"26208f8a-7662-410e-b91c-96797f43fe35\" (UID: \"26208f8a-7662-410e-b91c-96797f43fe35\") "
Sep 09 03:31:23.789871 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.789735    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpdn7\" (UniqueName: \"kubernetes.io/projected/26208f8a-7662-410e-b91c-96797f43fe35-kube-api-access-fpdn7\") pod \"26208f8a-7662-410e-b91c-96797f43fe35\" (UID: \"26208f8a-7662-410e-b91c-96797f43fe35\") "
Sep 09 03:31:23.789871 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.789841    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26208f8a-7662-410e-b91c-96797f43fe35-openshift-service-ca\") pod \"26208f8a-7662-410e-b91c-96797f43fe35\" (UID: \"26208f8a-7662-410e-b91c-96797f43fe35\") "
Sep 09 03:31:23.790045 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.790020    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26208f8a-7662-410e-b91c-96797f43fe35-maas-api-tls\") pod \"26208f8a-7662-410e-b91c-96797f43fe35\" (UID: \"26208f8a-7662-410e-b91c-96797f43fe35\") "
Sep 09 03:31:23.790177 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.790099    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26208f8a-7662-410e-b91c-96797f43fe35-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "26208f8a-7662-410e-b91c-96797f43fe35" (UID: "26208f8a-7662-410e-b91c-96797f43fe35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:31:23.790391 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.790367    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26208f8a-7662-410e-b91c-96797f43fe35-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:31:23.791841 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.791817    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26208f8a-7662-410e-b91c-96797f43fe35-kube-api-access-fpdn7" (OuterVolumeSpecName: "kube-api-access-fpdn7") pod "26208f8a-7662-410e-b91c-96797f43fe35" (UID: "26208f8a-7662-410e-b91c-96797f43fe35"). InnerVolumeSpecName "kube-api-access-fpdn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:31:23.791940 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.791884    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26208f8a-7662-410e-b91c-96797f43fe35-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "26208f8a-7662-410e-b91c-96797f43fe35" (UID: "26208f8a-7662-410e-b91c-96797f43fe35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:31:23.791987 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.791936    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26208f8a-7662-410e-b91c-96797f43fe35-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "26208f8a-7662-410e-b91c-96797f43fe35" (UID: "26208f8a-7662-410e-b91c-96797f43fe35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:31:23.799051 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.799032    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:23.799137 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.799071    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:23.807040 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.807015    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:23.811506 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.811488    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:23.891358 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.891322    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26208f8a-7662-410e-b91c-96797f43fe35-metrics-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:31:23.891358 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.891353    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpdn7\" (UniqueName: \"kubernetes.io/projected/26208f8a-7662-410e-b91c-96797f43fe35-kube-api-access-fpdn7\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:31:23.891552 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:23.891368    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26208f8a-7662-410e-b91c-96797f43fe35-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:31:24.452469 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:24.452432    2560 generic.go:356] "Generic (PLEG): container finished" podID="26208f8a-7662-410e-b91c-96797f43fe35" containerID="d21fffb154462b0b3b1fd387edd5ac67ce9e6055c08163fab6734a42621e7dca" exitCode=0
Sep 09 03:31:24.452661 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:24.452478    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"
Sep 09 03:31:24.452661 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:24.452517    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq" event={"ID":"26208f8a-7662-410e-b91c-96797f43fe35","Type":"ContainerDied","Data":"d21fffb154462b0b3b1fd387edd5ac67ce9e6055c08163fab6734a42621e7dca"}
Sep 09 03:31:24.452661 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:24.452569    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq" event={"ID":"26208f8a-7662-410e-b91c-96797f43fe35","Type":"ContainerDied","Data":"360ed115b2c2634880a3c5928ec924a3d42992be601bc709ddf8f69cd41748b5"}
Sep 09 03:31:24.452661 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:24.452589    2560 scope.go:117] "RemoveContainer" containerID="d21fffb154462b0b3b1fd387edd5ac67ce9e6055c08163fab6734a42621e7dca"
Sep 09 03:31:24.464266 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:24.464240    2560 scope.go:117] "RemoveContainer" containerID="d21fffb154462b0b3b1fd387edd5ac67ce9e6055c08163fab6734a42621e7dca"
Sep 09 03:31:24.464699 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:31:24.464678    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21fffb154462b0b3b1fd387edd5ac67ce9e6055c08163fab6734a42621e7dca\": container with ID starting with d21fffb154462b0b3b1fd387edd5ac67ce9e6055c08163fab6734a42621e7dca not found: ID does not exist" containerID="d21fffb154462b0b3b1fd387edd5ac67ce9e6055c08163fab6734a42621e7dca"
Sep 09 03:31:24.464792 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:24.464705    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21fffb154462b0b3b1fd387edd5ac67ce9e6055c08163fab6734a42621e7dca"} err="failed to get container status \"d21fffb154462b0b3b1fd387edd5ac67ce9e6055c08163fab6734a42621e7dca\": rpc error: code = NotFound desc = could not find container \"d21fffb154462b0b3b1fd387edd5ac67ce9e6055c08163fab6734a42621e7dca\": container with ID starting with d21fffb154462b0b3b1fd387edd5ac67ce9e6055c08163fab6734a42621e7dca not found: ID does not exist"
Sep 09 03:31:24.465876 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:24.465855    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:31:24.475674 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:24.475653    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"]
Sep 09 03:31:24.480510 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:24.480485    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0f5d040-749759c6b-bc9bq"]
Sep 09 03:31:24.942961 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:24.942932    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"]
Sep 09 03:31:24.943151 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:24.943130    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4" podUID="1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" containerName="istio-proxy" containerID="cri-o://a105ccdea25eecda6b78f4ea39c114086a2fb3b8c63a639d7a182f1c37c21853" gracePeriod=30
Sep 09 03:31:24.975245 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:24.975218    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:25.359051 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.359019    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26208f8a-7662-410e-b91c-96797f43fe35" path="/var/lib/kubelet/pods/26208f8a-7662-410e-b91c-96797f43fe35/volumes"
Sep 09 03:31:25.424811 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.424778    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:25.734367 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.734280    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"]
Sep 09 03:31:25.734770 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.734756    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26208f8a-7662-410e-b91c-96797f43fe35" containerName="maas-api"
Sep 09 03:31:25.734814 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.734772    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="26208f8a-7662-410e-b91c-96797f43fe35" containerName="maas-api"
Sep 09 03:31:25.734883 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.734871    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="26208f8a-7662-410e-b91c-96797f43fe35" containerName="maas-api"
Sep 09 03:31:25.739157 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.739137    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.743057 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.743035    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ad343832-openshift-default-dockercfg-jx4l2\""
Sep 09 03:31:25.760299 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.760279    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"]
Sep 09 03:31:25.784089 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.784057    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:25.811706 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.811677    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-envoy\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.811823 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.811726    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-workload-certs\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.811823 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.811786    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsll4\" (UniqueName: \"kubernetes.io/projected/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-kube-api-access-xsll4\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.811923 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.811844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-workload-socket\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.811923 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.811893    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-data\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.812010 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.811935    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-credential-socket\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.812010 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.811952    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-token\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.812010 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.811967    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istiod-ca-cert\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.812124 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.812033    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-podinfo\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.913516 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.913484    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-credential-socket\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.913516 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.913515    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-token\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.913775 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.913533    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istiod-ca-cert\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.913775 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.913572    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-podinfo\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.913775 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.913684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-envoy\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.913775 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.913721    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-workload-certs\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.913775 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.913760    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsll4\" (UniqueName: \"kubernetes.io/projected/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-kube-api-access-xsll4\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.914057 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.913810    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-workload-socket\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.914057 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.913841    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-data\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.914057 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.913968    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-credential-socket\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.914226 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.914100    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-workload-certs\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.914282 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.914261    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-data\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.914341 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.914323    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istiod-ca-cert\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.914404 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.914352    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-workload-socket\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.916141 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.916121    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-envoy\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.916340 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.916323    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-podinfo\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.923454 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.923434    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-token\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:25.923700 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:25.923682    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsll4\" (UniqueName: \"kubernetes.io/projected/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-kube-api-access-xsll4\") pod \"e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") " pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:26.051023 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:26.050932    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:26.369722 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:26.369692    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"]
Sep 09 03:31:26.370394 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:31:26.370358    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e25da1d_fa11_44f6_9c4b_38b38cfc8cb4.slice/crio-92a1faa453b616490b75323b21da10e60afffb2d6435b51567a8a70a79c2b0fd WatchSource:0}: Error finding container 92a1faa453b616490b75323b21da10e60afffb2d6435b51567a8a70a79c2b0fd: Status 404 returned error can't find the container with id 92a1faa453b616490b75323b21da10e60afffb2d6435b51567a8a70a79c2b0fd
Sep 09 03:31:26.372940 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:26.372904    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:31:26.373006 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:26.372986    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:31:26.373047 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:26.373030    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:31:26.409085 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:26.409060    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:26.467260 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:26.467227    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v" event={"ID":"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4","Type":"ContainerStarted","Data":"82b84ea72d99862ed0326ea41adc6795b3e4da2d46ca715b7e0717eac1859c42"}
Sep 09 03:31:26.467399 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:26.467330    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v" event={"ID":"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4","Type":"ContainerStarted","Data":"92a1faa453b616490b75323b21da10e60afffb2d6435b51567a8a70a79c2b0fd"}
Sep 09 03:31:26.504553 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:26.504499    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v" podStartSLOduration=1.504479356 podStartE2EDuration="1.504479356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:31:26.502409177 +0000 UTC m=+1103.639157041" watchObservedRunningTime="2026-09-09 03:31:26.504479356 +0000 UTC m=+1103.641227222"
Sep 09 03:31:26.834310 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:26.834224    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:27.051110 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:27.051079    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:27.055981 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:27.055960    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:27.472734 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:27.472705    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:27.473632 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:27.473612    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:27.886846 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:27.886819    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:27.991099 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:27.991065    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:29.069916 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:29.069886    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:29.165121 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:29.165079    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:29.304330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:29.304278    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:29.683185 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:29.683141    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:29.789903 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:29.789864    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:30.195471 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.195446    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:31:30.359587 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.359560    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-data\") pod \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") "
Sep 09 03:31:30.359798 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.359642    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-token\") pod \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") "
Sep 09 03:31:30.359798 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.359679    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blnsq\" (UniqueName: \"kubernetes.io/projected/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-kube-api-access-blnsq\") pod \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") "
Sep 09 03:31:30.359798 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.359718    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-workload-socket\") pod \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") "
Sep 09 03:31:30.359798 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.359755    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-envoy\") pod \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") "
Sep 09 03:31:30.359798 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.359785    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-credential-socket\") pod \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") "
Sep 09 03:31:30.360059 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.359849    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-podinfo\") pod \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") "
Sep 09 03:31:30.360059 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.359866    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" (UID: "1a8a1cf2-1fcf-4910-9e69-de13e7342b4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:31:30.360059 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.359887    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istiod-ca-cert\") pod \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") "
Sep 09 03:31:30.360059 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.359921    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-workload-certs\") pod \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\" (UID: \"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d\") "
Sep 09 03:31:30.360059 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.360013    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" (UID: "1a8a1cf2-1fcf-4910-9e69-de13e7342b4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:31:30.360305 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.360108    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" (UID: "1a8a1cf2-1fcf-4910-9e69-de13e7342b4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:31:30.360360 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.360329    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" (UID: "1a8a1cf2-1fcf-4910-9e69-de13e7342b4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:31:30.360436 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.360348    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" (UID: "1a8a1cf2-1fcf-4910-9e69-de13e7342b4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:31:30.360436 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.360357    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-credential-socket\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:31:30.360436 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.360402    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-data\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:31:30.360436 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.360420    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-workload-socket\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:31:30.362131 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.362102    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" (UID: "1a8a1cf2-1fcf-4910-9e69-de13e7342b4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:31:30.362260 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.362107    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-kube-api-access-blnsq" (OuterVolumeSpecName: "kube-api-access-blnsq") pod "1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" (UID: "1a8a1cf2-1fcf-4910-9e69-de13e7342b4d"). InnerVolumeSpecName "kube-api-access-blnsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:31:30.362260 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.362130    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" (UID: "1a8a1cf2-1fcf-4910-9e69-de13e7342b4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:31:30.362384 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.362356    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" (UID: "1a8a1cf2-1fcf-4910-9e69-de13e7342b4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:31:30.461605 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.461574    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-token\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:31:30.461752 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.461617    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blnsq\" (UniqueName: \"kubernetes.io/projected/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-kube-api-access-blnsq\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:31:30.461752 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.461629    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-envoy\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:31:30.461752 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.461638    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istio-podinfo\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:31:30.461752 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.461646    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-istiod-ca-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:31:30.461752 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.461653    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d-workload-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:31:30.489658 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.489623    2560 generic.go:356] "Generic (PLEG): container finished" podID="1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" containerID="a105ccdea25eecda6b78f4ea39c114086a2fb3b8c63a639d7a182f1c37c21853" exitCode=0
Sep 09 03:31:30.489820 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.489674    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"
Sep 09 03:31:30.489820 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.489714    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4" event={"ID":"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d","Type":"ContainerDied","Data":"a105ccdea25eecda6b78f4ea39c114086a2fb3b8c63a639d7a182f1c37c21853"}
Sep 09 03:31:30.489820 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.489780    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4" event={"ID":"1a8a1cf2-1fcf-4910-9e69-de13e7342b4d","Type":"ContainerDied","Data":"662fdf67db7816b79e9e551ba2efcf10385f5cf7d1d55f73271f5d1c36a2adff"}
Sep 09 03:31:30.489820 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.489813    2560 scope.go:117] "RemoveContainer" containerID="a105ccdea25eecda6b78f4ea39c114086a2fb3b8c63a639d7a182f1c37c21853"
Sep 09 03:31:30.499019 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.498995    2560 scope.go:117] "RemoveContainer" containerID="b241a72486fbb6ff364bc4a5e85299aa48897305241dcd51a8d2a0cea506c237"
Sep 09 03:31:30.506931 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.506917    2560 scope.go:117] "RemoveContainer" containerID="a105ccdea25eecda6b78f4ea39c114086a2fb3b8c63a639d7a182f1c37c21853"
Sep 09 03:31:30.507185 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:31:30.507169    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a105ccdea25eecda6b78f4ea39c114086a2fb3b8c63a639d7a182f1c37c21853\": container with ID starting with a105ccdea25eecda6b78f4ea39c114086a2fb3b8c63a639d7a182f1c37c21853 not found: ID does not exist" containerID="a105ccdea25eecda6b78f4ea39c114086a2fb3b8c63a639d7a182f1c37c21853"
Sep 09 03:31:30.507228 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.507194    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a105ccdea25eecda6b78f4ea39c114086a2fb3b8c63a639d7a182f1c37c21853"} err="failed to get container status \"a105ccdea25eecda6b78f4ea39c114086a2fb3b8c63a639d7a182f1c37c21853\": rpc error: code = NotFound desc = could not find container \"a105ccdea25eecda6b78f4ea39c114086a2fb3b8c63a639d7a182f1c37c21853\": container with ID starting with a105ccdea25eecda6b78f4ea39c114086a2fb3b8c63a639d7a182f1c37c21853 not found: ID does not exist"
Sep 09 03:31:30.507228 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.507209    2560 scope.go:117] "RemoveContainer" containerID="b241a72486fbb6ff364bc4a5e85299aa48897305241dcd51a8d2a0cea506c237"
Sep 09 03:31:30.507427 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:31:30.507410    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b241a72486fbb6ff364bc4a5e85299aa48897305241dcd51a8d2a0cea506c237\": container with ID starting with b241a72486fbb6ff364bc4a5e85299aa48897305241dcd51a8d2a0cea506c237 not found: ID does not exist" containerID="b241a72486fbb6ff364bc4a5e85299aa48897305241dcd51a8d2a0cea506c237"
Sep 09 03:31:30.507460 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.507432    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b241a72486fbb6ff364bc4a5e85299aa48897305241dcd51a8d2a0cea506c237"} err="failed to get container status \"b241a72486fbb6ff364bc4a5e85299aa48897305241dcd51a8d2a0cea506c237\": rpc error: code = NotFound desc = could not find container \"b241a72486fbb6ff364bc4a5e85299aa48897305241dcd51a8d2a0cea506c237\": container with ID starting with b241a72486fbb6ff364bc4a5e85299aa48897305241dcd51a8d2a0cea506c237 not found: ID does not exist"
Sep 09 03:31:30.514452 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.514428    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"]
Sep 09 03:31:30.520170 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:30.520150    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b0f5d040-openshift-default-546f8fcd58-j9vf4"]
Sep 09 03:31:31.359170 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:31.359133    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" path="/var/lib/kubelet/pods/1a8a1cf2-1fcf-4910-9e69-de13e7342b4d/volumes"
Sep 09 03:31:35.875469 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.875429    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"]
Sep 09 03:31:35.876096 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.876073    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" containerName="istio-proxy"
Sep 09 03:31:35.876162 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.876100    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" containerName="istio-proxy"
Sep 09 03:31:35.876162 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.876114    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" containerName="istio-proxy"
Sep 09 03:31:35.876162 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.876122    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" containerName="istio-proxy"
Sep 09 03:31:35.876323 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.876231    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" containerName="istio-proxy"
Sep 09 03:31:35.876323 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.876244    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8a1cf2-1fcf-4910-9e69-de13e7342b4d" containerName="istio-proxy"
Sep 09 03:31:35.879843 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.879822    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:35.883756 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.883733    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-40a1c9\"/\"test-model-40a1c9-kserve-self-signed-certs\""
Sep 09 03:31:35.883975 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.883734    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-40a1c9\"/\"default-dockercfg-cf6t6\""
Sep 09 03:31:35.883975 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.883735    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-40a1c9\"/\"openshift-service-ca.crt\""
Sep 09 03:31:35.884153 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.883751    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-40a1c9\"/\"kube-root-ca.crt\""
Sep 09 03:31:35.889064 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.889042    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"]
Sep 09 03:31:35.910518 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.910488    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-tmp-dir\") pod \"test-model-40a1c9-kserve-79cdb4454f-bj9kg\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") " pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:35.910669 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.910560    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-dshm\") pod \"test-model-40a1c9-kserve-79cdb4454f-bj9kg\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") " pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:35.910669 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.910624    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8958171-2e22-4a18-bdc6-5df916b64058-tls-certs\") pod \"test-model-40a1c9-kserve-79cdb4454f-bj9kg\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") " pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:35.910825 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.910713    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-home\") pod \"test-model-40a1c9-kserve-79cdb4454f-bj9kg\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") " pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:35.910825 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.910776    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-model-cache\") pod \"test-model-40a1c9-kserve-79cdb4454f-bj9kg\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") " pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:35.910825 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.910794    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x27gz\" (UniqueName: \"kubernetes.io/projected/e8958171-2e22-4a18-bdc6-5df916b64058-kube-api-access-x27gz\") pod \"test-model-40a1c9-kserve-79cdb4454f-bj9kg\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") " pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:35.911245 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:35.911222    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:36.011536 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:36.011499    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-home\") pod \"test-model-40a1c9-kserve-79cdb4454f-bj9kg\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") " pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:36.011724 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:36.011610    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-model-cache\") pod \"test-model-40a1c9-kserve-79cdb4454f-bj9kg\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") " pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:36.011724 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:36.011637    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x27gz\" (UniqueName: \"kubernetes.io/projected/e8958171-2e22-4a18-bdc6-5df916b64058-kube-api-access-x27gz\") pod \"test-model-40a1c9-kserve-79cdb4454f-bj9kg\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") " pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:36.011724 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:36.011686    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-tmp-dir\") pod \"test-model-40a1c9-kserve-79cdb4454f-bj9kg\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") " pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:36.011724 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:36.011714    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-dshm\") pod \"test-model-40a1c9-kserve-79cdb4454f-bj9kg\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") " pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:36.011942 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:36.011741    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8958171-2e22-4a18-bdc6-5df916b64058-tls-certs\") pod \"test-model-40a1c9-kserve-79cdb4454f-bj9kg\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") " pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:36.012005 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:36.011988    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-home\") pod \"test-model-40a1c9-kserve-79cdb4454f-bj9kg\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") " pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:36.012060 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:36.012044    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-model-cache\") pod \"test-model-40a1c9-kserve-79cdb4454f-bj9kg\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") " pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:36.012111 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:36.012079    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-tmp-dir\") pod \"test-model-40a1c9-kserve-79cdb4454f-bj9kg\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") " pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:36.013814 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:36.013788    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-dshm\") pod \"test-model-40a1c9-kserve-79cdb4454f-bj9kg\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") " pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:36.014278 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:36.014259    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8958171-2e22-4a18-bdc6-5df916b64058-tls-certs\") pod \"test-model-40a1c9-kserve-79cdb4454f-bj9kg\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") " pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:36.024865 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:36.024844    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x27gz\" (UniqueName: \"kubernetes.io/projected/e8958171-2e22-4a18-bdc6-5df916b64058-kube-api-access-x27gz\") pod \"test-model-40a1c9-kserve-79cdb4454f-bj9kg\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") " pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:36.191752 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:36.191657    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:36.468581 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:36.468556    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"]
Sep 09 03:31:36.470694 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:31:36.470672    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8958171_2e22_4a18_bdc6_5df916b64058.slice/crio-afa84eac0c0d959078d9142b3cfbd5ddbaf7e0893822a22b32fa887810e7c361 WatchSource:0}: Error finding container afa84eac0c0d959078d9142b3cfbd5ddbaf7e0893822a22b32fa887810e7c361: Status 404 returned error can't find the container with id afa84eac0c0d959078d9142b3cfbd5ddbaf7e0893822a22b32fa887810e7c361
Sep 09 03:31:36.521959 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:36.521931    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg" event={"ID":"e8958171-2e22-4a18-bdc6-5df916b64058","Type":"ContainerStarted","Data":"afa84eac0c0d959078d9142b3cfbd5ddbaf7e0893822a22b32fa887810e7c361"}
Sep 09 03:31:36.992272 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:36.992237    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:31:37.528386 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:37.528351    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg" event={"ID":"e8958171-2e22-4a18-bdc6-5df916b64058","Type":"ContainerStarted","Data":"21b258b74932d2df96463d9248d9a41b97402582914e358f3bb64c1c1408cb3f"}
Sep 09 03:31:37.550193 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:37.550142    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg" podStartSLOduration=2.550126178 podStartE2EDuration="2.550126178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:31:37.548583995 +0000 UTC m=+1114.685331859" watchObservedRunningTime="2026-09-09 03:31:37.550126178 +0000 UTC m=+1114.686874043"
Sep 09 03:31:46.192270 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:46.192226    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:46.192772 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:46.192303    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:46.205297 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:46.205267    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:46.578267 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:46.578238    2560 generic.go:356] "Generic (PLEG): container finished" podID="1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" containerID="82b84ea72d99862ed0326ea41adc6795b3e4da2d46ca715b7e0717eac1859c42" exitCode=0
Sep 09 03:31:46.578473 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:46.578314    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v" event={"ID":"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4","Type":"ContainerDied","Data":"82b84ea72d99862ed0326ea41adc6795b3e4da2d46ca715b7e0717eac1859c42"}
Sep 09 03:31:46.578841 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:46.578822    2560 scope.go:117] "RemoveContainer" containerID="82b84ea72d99862ed0326ea41adc6795b3e4da2d46ca715b7e0717eac1859c42"
Sep 09 03:31:46.579869 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:46.579843    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:31:46.579935 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:46.579917    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:31:46.579975 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:46.579947    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 03:31:46.590300 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:46.590281    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:31:47.587472 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:47.587432    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v" event={"ID":"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4","Type":"ContainerStarted","Data":"628542275f92827b65e39baf3a757685bbdd7d4f7c6d8da4448d979120e57c1b"}
Sep 09 03:31:48.051522 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:48.051485    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:48.051744 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:48.051687    2560 patch_prober.go:28] interesting pod/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 09 03:31:48.051815 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:48.051741    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v" podUID="1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 09 03:31:48.435526 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:48.435494    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a2fc47-lpff5"]
Sep 09 03:31:48.439029 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:48.439008    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a2fc47-lpff5"
Sep 09 03:31:48.441589 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:48.441567    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7q9f\""
Sep 09 03:31:48.447093 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:48.447004    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a2fc47-lpff5"]
Sep 09 03:31:48.537660 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:48.537588    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f37b0f12-708a-4e5f-981f-553d54c963d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a2fc47-lpff5\" (UID: \"f37b0f12-708a-4e5f-981f-553d54c963d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a2fc47-lpff5"
Sep 09 03:31:48.548863 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:48.548835    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:48.638796 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:48.638760    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f37b0f12-708a-4e5f-981f-553d54c963d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a2fc47-lpff5\" (UID: \"f37b0f12-708a-4e5f-981f-553d54c963d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a2fc47-lpff5"
Sep 09 03:31:48.639421 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:48.639400    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f37b0f12-708a-4e5f-981f-553d54c963d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a2fc47-lpff5\" (UID: \"f37b0f12-708a-4e5f-981f-553d54c963d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a2fc47-lpff5"
Sep 09 03:31:48.750655 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:48.750540    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a2fc47-lpff5"
Sep 09 03:31:49.045134 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:49.045105    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a2fc47-lpff5"]
Sep 09 03:31:49.047272 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:31:49.047241    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf37b0f12_708a_4e5f_981f_553d54c963d1.slice/crio-7fe7b183fb8543e2d9ae2d922462e39a444c49dfddb47c190926674bd8f4962a WatchSource:0}: Error finding container 7fe7b183fb8543e2d9ae2d922462e39a444c49dfddb47c190926674bd8f4962a: Status 404 returned error can't find the container with id 7fe7b183fb8543e2d9ae2d922462e39a444c49dfddb47c190926674bd8f4962a
Sep 09 03:31:49.051262 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:49.051235    2560 patch_prober.go:28] interesting pod/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 09 03:31:49.051352 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:49.051283    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v" podUID="1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 09 03:31:49.601947 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:49.601900    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a2fc47-lpff5" event={"ID":"f37b0f12-708a-4e5f-981f-553d54c963d1","Type":"ContainerStarted","Data":"b3eab3861ae34839e63f443684bb4faf403069f80c2d2175d1cda7cb63500b9c"}
Sep 09 03:31:49.602123 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:49.601972    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a2fc47-lpff5" event={"ID":"f37b0f12-708a-4e5f-981f-553d54c963d1","Type":"ContainerStarted","Data":"7fe7b183fb8543e2d9ae2d922462e39a444c49dfddb47c190926674bd8f4962a"}
Sep 09 03:31:49.618556 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:49.618509    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a2fc47-lpff5" podStartSLOduration=1.618494601 podStartE2EDuration="1.618494601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:31:49.616651166 +0000 UTC m=+1126.753399033" watchObservedRunningTime="2026-09-09 03:31:49.618494601 +0000 UTC m=+1126.755242466"
Sep 09 03:31:50.051760 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:50.051727    2560 patch_prober.go:28] interesting pod/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 09 03:31:50.052143 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:50.051783    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v" podUID="1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 09 03:31:50.129739 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:50.129707    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:50.489483 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:50.489442    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:31:50.609758 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:50.609720    2560 generic.go:356] "Generic (PLEG): container finished" podID="f37b0f12-708a-4e5f-981f-553d54c963d1" containerID="b3eab3861ae34839e63f443684bb4faf403069f80c2d2175d1cda7cb63500b9c" exitCode=0
Sep 09 03:31:50.609912 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:50.609813    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a2fc47-lpff5" event={"ID":"f37b0f12-708a-4e5f-981f-553d54c963d1","Type":"ContainerDied","Data":"b3eab3861ae34839e63f443684bb4faf403069f80c2d2175d1cda7cb63500b9c"}
Sep 09 03:31:51.051949 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:51.051903    2560 patch_prober.go:28] interesting pod/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 09 03:31:51.052429 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:51.051982    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v" podUID="1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 09 03:31:51.792281 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:51.792255    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a2fc47-lpff5"
Sep 09 03:31:51.876675 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:51.876064    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f37b0f12-708a-4e5f-981f-553d54c963d1-maas-api-service-ca\") pod \"f37b0f12-708a-4e5f-981f-553d54c963d1\" (UID: \"f37b0f12-708a-4e5f-981f-553d54c963d1\") "
Sep 09 03:31:51.876870 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:51.876757    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f37b0f12-708a-4e5f-981f-553d54c963d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f37b0f12-708a-4e5f-981f-553d54c963d1" (UID: "f37b0f12-708a-4e5f-981f-553d54c963d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:31:51.977100 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:51.977051    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f37b0f12-708a-4e5f-981f-553d54c963d1-maas-api-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:31:52.052276 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:52.052236    2560 patch_prober.go:28] interesting pod/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 09 03:31:52.052690 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:52.052300    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v" podUID="1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 09 03:31:52.485520 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:52.485488    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:52.622907 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:52.622876    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a2fc47-lpff5" event={"ID":"f37b0f12-708a-4e5f-981f-553d54c963d1","Type":"ContainerDied","Data":"7fe7b183fb8543e2d9ae2d922462e39a444c49dfddb47c190926674bd8f4962a"}
Sep 09 03:31:52.622907 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:52.622896    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a2fc47-lpff5"
Sep 09 03:31:52.623108 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:52.622929    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fe7b183fb8543e2d9ae2d922462e39a444c49dfddb47c190926674bd8f4962a"
Sep 09 03:31:52.759516 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:52.759424    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ad343832-wqhq2"]
Sep 09 03:31:52.759987 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:52.759969    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f37b0f12-708a-4e5f-981f-553d54c963d1" containerName="revoke-keys"
Sep 09 03:31:52.759987 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:52.759988    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f37b0f12-708a-4e5f-981f-553d54c963d1" containerName="revoke-keys"
Sep 09 03:31:52.760163 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:52.760076    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f37b0f12-708a-4e5f-981f-553d54c963d1" containerName="revoke-keys"
Sep 09 03:31:52.763296 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:52.763275    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ad343832-wqhq2"
Sep 09 03:31:52.766505 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:52.766483    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7q9f\""
Sep 09 03:31:52.773558 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:52.773529    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ad343832-wqhq2"]
Sep 09 03:31:52.888721 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:52.888673    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ad343832-wqhq2\" (UID: \"67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ad343832-wqhq2"
Sep 09 03:31:52.989810 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:52.989773    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ad343832-wqhq2\" (UID: \"67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ad343832-wqhq2"
Sep 09 03:31:52.990812 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:52.990782    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ad343832-wqhq2\" (UID: \"67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ad343832-wqhq2"
Sep 09 03:31:53.075110 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:53.075007    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ad343832-wqhq2"
Sep 09 03:31:53.077274 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:53.077245    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:53.077687 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:53.077659    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:53.078861 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:53.078813    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:31:53.366758 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:31:53.366721    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a3f2b9_c6b0_4ca8_bf67_22247ffeffe3.slice/crio-8de663e55267bc088f83343ffa9e10ca4bd75d69c5f359f107b42a356175fe10 WatchSource:0}: Error finding container 8de663e55267bc088f83343ffa9e10ca4bd75d69c5f359f107b42a356175fe10: Status 404 returned error can't find the container with id 8de663e55267bc088f83343ffa9e10ca4bd75d69c5f359f107b42a356175fe10
Sep 09 03:31:53.368530 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:53.368504    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ad343832-wqhq2"]
Sep 09 03:31:53.631011 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:53.630903    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ad343832-wqhq2" event={"ID":"67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3","Type":"ContainerStarted","Data":"04d3c55f7c155dd38c9dea5b1aacdcad8d107e6da4410a1b1ce6d97dc96ea39a"}
Sep 09 03:31:53.631268 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:53.631235    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ad343832-wqhq2" event={"ID":"67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3","Type":"ContainerStarted","Data":"8de663e55267bc088f83343ffa9e10ca4bd75d69c5f359f107b42a356175fe10"}
Sep 09 03:31:53.660051 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:53.660005    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ad343832-wqhq2" podStartSLOduration=1.659989588 podStartE2EDuration="1.659989588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:31:53.65778632 +0000 UTC m=+1130.794534187" watchObservedRunningTime="2026-09-09 03:31:53.659989588 +0000 UTC m=+1130.796737497"
Sep 09 03:31:54.341945 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:54.341841    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"]
Sep 09 03:31:54.365049 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:54.365019    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:54.638559 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:54.638526    2560 generic.go:356] "Generic (PLEG): container finished" podID="67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3" containerID="04d3c55f7c155dd38c9dea5b1aacdcad8d107e6da4410a1b1ce6d97dc96ea39a" exitCode=0
Sep 09 03:31:54.638725 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:54.638634    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ad343832-wqhq2" event={"ID":"67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3","Type":"ContainerDied","Data":"04d3c55f7c155dd38c9dea5b1aacdcad8d107e6da4410a1b1ce6d97dc96ea39a"}
Sep 09 03:31:55.777238 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:55.777211    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ad343832-wqhq2"
Sep 09 03:31:55.916952 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:55.916846    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3-maas-api-service-ca\") pod \"67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3\" (UID: \"67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3\") "
Sep 09 03:31:55.917269 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:55.917234    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3" (UID: "67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:31:56.017967 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:56.017930    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3-maas-api-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:31:56.650702 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:56.650672    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ad343832-wqhq2"
Sep 09 03:31:56.650868 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:56.650714    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ad343832-wqhq2" event={"ID":"67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3","Type":"ContainerDied","Data":"8de663e55267bc088f83343ffa9e10ca4bd75d69c5f359f107b42a356175fe10"}
Sep 09 03:31:56.650868 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:56.650761    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8de663e55267bc088f83343ffa9e10ca4bd75d69c5f359f107b42a356175fe10"
Sep 09 03:31:58.646324 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:58.646293    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:59.290281 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:59.290239    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:59.394695 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:59.394664    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:59.484558 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:59.484522    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-a2fc47-29815410-qm65d"]
Sep 09 03:31:59.490452 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:59.490401    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:59.491263 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:59.491242    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-a2fc47-29815410-qm65d"]
Sep 09 03:31:59.556297 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:59.556210    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ad343832-wqhq2"]
Sep 09 03:31:59.560318 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:59.560294    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ad343832-wqhq2"]
Sep 09 03:31:59.580316 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:59.580278    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:59.653866 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:59.653828    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:59.695190 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:59.695163    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"]
Sep 09 03:31:59.695418 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:59.695393    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v" podUID="1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" containerName="istio-proxy" containerID="cri-o://628542275f92827b65e39baf3a757685bbdd7d4f7c6d8da4448d979120e57c1b" gracePeriod=30
Sep 09 03:31:59.772182 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:59.772153    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:31:59.845914 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:59.845828    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a2fc47-lpff5"]
Sep 09 03:31:59.847715 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:59.847695    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a2fc47-lpff5"]
Sep 09 03:31:59.861942 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:31:59.861910    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:00.244062 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:00.244024    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:00.427063 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:00.427025    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:00.526397 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:00.526299    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:00.628987 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:00.628950    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:01.359441 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:01.359404    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3" path="/var/lib/kubelet/pods/67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3/volumes"
Sep 09 03:32:01.359914 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:01.359745    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db309424-3aad-447e-8700-0da729a89d17" path="/var/lib/kubelet/pods/db309424-3aad-447e-8700-0da729a89d17/volumes"
Sep 09 03:32:01.360001 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:01.359987    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f37b0f12-708a-4e5f-981f-553d54c963d1" path="/var/lib/kubelet/pods/f37b0f12-708a-4e5f-981f-553d54c963d1/volumes"
Sep 09 03:32:04.940231 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:04.940201    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:32:05.111588 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.111554    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-workload-certs\") pod \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") "
Sep 09 03:32:05.111780 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.111619    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsll4\" (UniqueName: \"kubernetes.io/projected/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-kube-api-access-xsll4\") pod \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") "
Sep 09 03:32:05.111780 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.111658    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-credential-socket\") pod \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") "
Sep 09 03:32:05.111780 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.111673    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istiod-ca-cert\") pod \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") "
Sep 09 03:32:05.111780 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.111696    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-envoy\") pod \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") "
Sep 09 03:32:05.111994 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.111852    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-token\") pod \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") "
Sep 09 03:32:05.111994 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.111901    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" (UID: "1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:05.111994 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.111911    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-workload-socket\") pod \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") "
Sep 09 03:32:05.111994 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.111969    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-data\") pod \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") "
Sep 09 03:32:05.112209 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.111925    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" (UID: "1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:05.112209 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.112064    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-podinfo\") pod \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\" (UID: \"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4\") "
Sep 09 03:32:05.112209 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.112135    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" (UID: "1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:05.112374 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.112231    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-data" (OuterVolumeSpecName: "istio-data") pod "1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" (UID: "1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:05.112374 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.112287    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" (UID: "1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:32:05.112505 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.112466    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-workload-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:05.112505 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.112487    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-credential-socket\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:05.112505 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.112503    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istiod-ca-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:05.112667 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.112516    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-workload-socket\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:05.112667 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.112529    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-data\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:05.114105 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.114078    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-token" (OuterVolumeSpecName: "istio-token") pod "1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" (UID: "1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:05.114230 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.114170    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" (UID: "1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:32:05.114374 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.114350    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" (UID: "1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:05.114423 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.114400    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-kube-api-access-xsll4" (OuterVolumeSpecName: "kube-api-access-xsll4") pod "1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" (UID: "1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4"). InnerVolumeSpecName "kube-api-access-xsll4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:05.214053 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.214017    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-podinfo\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:05.214053 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.214047    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsll4\" (UniqueName: \"kubernetes.io/projected/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-kube-api-access-xsll4\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:05.214053 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.214056    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-envoy\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:05.214053 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.214065    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4-istio-token\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:05.309722 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.309688    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:05.701191 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.701112    2560 generic.go:356] "Generic (PLEG): container finished" podID="1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" containerID="628542275f92827b65e39baf3a757685bbdd7d4f7c6d8da4448d979120e57c1b" exitCode=0
Sep 09 03:32:05.701191 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.701157    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v" event={"ID":"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4","Type":"ContainerDied","Data":"628542275f92827b65e39baf3a757685bbdd7d4f7c6d8da4448d979120e57c1b"}
Sep 09 03:32:05.701191 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.701159    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"
Sep 09 03:32:05.701402 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.701200    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v" event={"ID":"1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4","Type":"ContainerDied","Data":"92a1faa453b616490b75323b21da10e60afffb2d6435b51567a8a70a79c2b0fd"}
Sep 09 03:32:05.701402 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.701222    2560 scope.go:117] "RemoveContainer" containerID="628542275f92827b65e39baf3a757685bbdd7d4f7c6d8da4448d979120e57c1b"
Sep 09 03:32:05.709972 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.709953    2560 scope.go:117] "RemoveContainer" containerID="82b84ea72d99862ed0326ea41adc6795b3e4da2d46ca715b7e0717eac1859c42"
Sep 09 03:32:05.718436 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.718326    2560 scope.go:117] "RemoveContainer" containerID="628542275f92827b65e39baf3a757685bbdd7d4f7c6d8da4448d979120e57c1b"
Sep 09 03:32:05.718701 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:32:05.718681    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"628542275f92827b65e39baf3a757685bbdd7d4f7c6d8da4448d979120e57c1b\": container with ID starting with 628542275f92827b65e39baf3a757685bbdd7d4f7c6d8da4448d979120e57c1b not found: ID does not exist" containerID="628542275f92827b65e39baf3a757685bbdd7d4f7c6d8da4448d979120e57c1b"
Sep 09 03:32:05.718770 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.718709    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"628542275f92827b65e39baf3a757685bbdd7d4f7c6d8da4448d979120e57c1b"} err="failed to get container status \"628542275f92827b65e39baf3a757685bbdd7d4f7c6d8da4448d979120e57c1b\": rpc error: code = NotFound desc = could not find container \"628542275f92827b65e39baf3a757685bbdd7d4f7c6d8da4448d979120e57c1b\": container with ID starting with 628542275f92827b65e39baf3a757685bbdd7d4f7c6d8da4448d979120e57c1b not found: ID does not exist"
Sep 09 03:32:05.718770 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.718727    2560 scope.go:117] "RemoveContainer" containerID="82b84ea72d99862ed0326ea41adc6795b3e4da2d46ca715b7e0717eac1859c42"
Sep 09 03:32:05.718979 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:32:05.718957    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82b84ea72d99862ed0326ea41adc6795b3e4da2d46ca715b7e0717eac1859c42\": container with ID starting with 82b84ea72d99862ed0326ea41adc6795b3e4da2d46ca715b7e0717eac1859c42 not found: ID does not exist" containerID="82b84ea72d99862ed0326ea41adc6795b3e4da2d46ca715b7e0717eac1859c42"
Sep 09 03:32:05.719031 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.718988    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b84ea72d99862ed0326ea41adc6795b3e4da2d46ca715b7e0717eac1859c42"} err="failed to get container status \"82b84ea72d99862ed0326ea41adc6795b3e4da2d46ca715b7e0717eac1859c42\": rpc error: code = NotFound desc = could not find container \"82b84ea72d99862ed0326ea41adc6795b3e4da2d46ca715b7e0717eac1859c42\": container with ID starting with 82b84ea72d99862ed0326ea41adc6795b3e4da2d46ca715b7e0717eac1859c42 not found: ID does not exist"
Sep 09 03:32:05.720260 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.720242    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"]
Sep 09 03:32:05.724543 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:05.724521    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ad343832-openshift-default-6675d8b6d4-t2x8v"]
Sep 09 03:32:06.417542 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:06.417504    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:07.359140 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:07.359106    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" path="/var/lib/kubelet/pods/1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4/volumes"
Sep 09 03:32:07.933933 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:07.933899    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"]
Sep 09 03:32:07.962082 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:07.962047    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:07.995404 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:07.995371    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:32:09.361625 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.361567    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4" podUID="bb93bc05-0816-4395-a76c-ced478c9c49c" containerName="main" containerID="cri-o://b1011a469b692db33222d9ac44c4f6865e2cf896d52fe96bfac5491e99949a87" gracePeriod=45
Sep 09 03:32:09.608607 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.608569    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:32:09.732217 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.732131    2560 generic.go:356] "Generic (PLEG): container finished" podID="bb93bc05-0816-4395-a76c-ced478c9c49c" containerID="b1011a469b692db33222d9ac44c4f6865e2cf896d52fe96bfac5491e99949a87" exitCode=0
Sep 09 03:32:09.732217 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.732178    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"
Sep 09 03:32:09.732395 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.732213    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4" event={"ID":"bb93bc05-0816-4395-a76c-ced478c9c49c","Type":"ContainerDied","Data":"b1011a469b692db33222d9ac44c4f6865e2cf896d52fe96bfac5491e99949a87"}
Sep 09 03:32:09.732395 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.732255    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4" event={"ID":"bb93bc05-0816-4395-a76c-ced478c9c49c","Type":"ContainerDied","Data":"88d88295a99ec4058b1a8779d5fd684156c95175a2c05b066d148bb97199b05b"}
Sep 09 03:32:09.732395 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.732274    2560 scope.go:117] "RemoveContainer" containerID="b1011a469b692db33222d9ac44c4f6865e2cf896d52fe96bfac5491e99949a87"
Sep 09 03:32:09.740687 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.740676    2560 scope.go:117] "RemoveContainer" containerID="b1011a469b692db33222d9ac44c4f6865e2cf896d52fe96bfac5491e99949a87"
Sep 09 03:32:09.740955 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:32:09.740938    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1011a469b692db33222d9ac44c4f6865e2cf896d52fe96bfac5491e99949a87\": container with ID starting with b1011a469b692db33222d9ac44c4f6865e2cf896d52fe96bfac5491e99949a87 not found: ID does not exist" containerID="b1011a469b692db33222d9ac44c4f6865e2cf896d52fe96bfac5491e99949a87"
Sep 09 03:32:09.741004 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.740963    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1011a469b692db33222d9ac44c4f6865e2cf896d52fe96bfac5491e99949a87"} err="failed to get container status \"b1011a469b692db33222d9ac44c4f6865e2cf896d52fe96bfac5491e99949a87\": rpc error: code = NotFound desc = could not find container \"b1011a469b692db33222d9ac44c4f6865e2cf896d52fe96bfac5491e99949a87\": container with ID starting with b1011a469b692db33222d9ac44c4f6865e2cf896d52fe96bfac5491e99949a87 not found: ID does not exist"
Sep 09 03:32:09.759308 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.759284    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-tmp-dir\") pod \"bb93bc05-0816-4395-a76c-ced478c9c49c\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") "
Sep 09 03:32:09.759406 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.759319    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-dshm\") pod \"bb93bc05-0816-4395-a76c-ced478c9c49c\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") "
Sep 09 03:32:09.759406 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.759338    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-home\") pod \"bb93bc05-0816-4395-a76c-ced478c9c49c\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") "
Sep 09 03:32:09.759406 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.759380    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-model-cache\") pod \"bb93bc05-0816-4395-a76c-ced478c9c49c\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") "
Sep 09 03:32:09.759572 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.759416    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s65b4\" (UniqueName: \"kubernetes.io/projected/bb93bc05-0816-4395-a76c-ced478c9c49c-kube-api-access-s65b4\") pod \"bb93bc05-0816-4395-a76c-ced478c9c49c\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") "
Sep 09 03:32:09.759572 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.759453    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb93bc05-0816-4395-a76c-ced478c9c49c-tls-certs\") pod \"bb93bc05-0816-4395-a76c-ced478c9c49c\" (UID: \"bb93bc05-0816-4395-a76c-ced478c9c49c\") "
Sep 09 03:32:09.759712 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.759579    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb93bc05-0816-4395-a76c-ced478c9c49c" (UID: "bb93bc05-0816-4395-a76c-ced478c9c49c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:09.759712 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.759652    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb93bc05-0816-4395-a76c-ced478c9c49c" (UID: "bb93bc05-0816-4395-a76c-ced478c9c49c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:09.759712 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.759654    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-home" (OuterVolumeSpecName: "home") pod "bb93bc05-0816-4395-a76c-ced478c9c49c" (UID: "bb93bc05-0816-4395-a76c-ced478c9c49c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:09.759884 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.759867    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-model-cache\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:09.759934 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.759890    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-tmp-dir\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:09.759934 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.759903    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-home\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:09.761545 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.761519    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb93bc05-0816-4395-a76c-ced478c9c49c-kube-api-access-s65b4" (OuterVolumeSpecName: "kube-api-access-s65b4") pod "bb93bc05-0816-4395-a76c-ced478c9c49c" (UID: "bb93bc05-0816-4395-a76c-ced478c9c49c"). InnerVolumeSpecName "kube-api-access-s65b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:09.761610 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.761568    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-dshm" (OuterVolumeSpecName: "dshm") pod "bb93bc05-0816-4395-a76c-ced478c9c49c" (UID: "bb93bc05-0816-4395-a76c-ced478c9c49c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:09.761610 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.761568    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb93bc05-0816-4395-a76c-ced478c9c49c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb93bc05-0816-4395-a76c-ced478c9c49c" (UID: "bb93bc05-0816-4395-a76c-ced478c9c49c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:32:09.860863 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.860830    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s65b4\" (UniqueName: \"kubernetes.io/projected/bb93bc05-0816-4395-a76c-ced478c9c49c-kube-api-access-s65b4\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:09.860863 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.860861    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb93bc05-0816-4395-a76c-ced478c9c49c-tls-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:09.861048 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:09.860874    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb93bc05-0816-4395-a76c-ced478c9c49c-dshm\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:10.056756 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:10.056719    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"]
Sep 09 03:32:10.058469 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:10.058446    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-a2fc47/autores-model-a2fc47-kserve-7b89b54b6c-wq5f4"]
Sep 09 03:32:11.358818 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:11.358779    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb93bc05-0816-4395-a76c-ced478c9c49c" path="/var/lib/kubelet/pods/bb93bc05-0816-4395-a76c-ced478c9c49c/volumes"
Sep 09 03:32:11.441502 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:11.441471    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:11.716357 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:11.716274    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:12.227453 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:12.227415    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:12.321986 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:12.321954    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:12.494979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:12.494888    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:12.626088 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:12.626057    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:15.041353 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:15.041318    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:15.109015 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:15.108980    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:18.398988 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:18.398949    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:18.506488 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:18.506455    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:18.606799 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:18.606757    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:18.699041 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:18.698943    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:19.175766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:19.175733    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:20.076494 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:20.076452    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:20.170637 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:20.170581    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:21.194209 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.194163    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"]
Sep 09 03:32:21.194797 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.194775    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb93bc05-0816-4395-a76c-ced478c9c49c" containerName="main"
Sep 09 03:32:21.194847 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.194801    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb93bc05-0816-4395-a76c-ced478c9c49c" containerName="main"
Sep 09 03:32:21.194847 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.194815    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3" containerName="revoke-keys"
Sep 09 03:32:21.194847 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.194824    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3" containerName="revoke-keys"
Sep 09 03:32:21.194941 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.194849    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" containerName="istio-proxy"
Sep 09 03:32:21.194941 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.194858    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" containerName="istio-proxy"
Sep 09 03:32:21.195007 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.194972    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb93bc05-0816-4395-a76c-ced478c9c49c" containerName="main"
Sep 09 03:32:21.195007 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.194987    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" containerName="istio-proxy"
Sep 09 03:32:21.195007 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.194997    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="67a3f2b9-c6b0-4ca8-bf67-22247ffeffe3" containerName="revoke-keys"
Sep 09 03:32:21.199973 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.199951    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:32:21.202671 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.202649    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-958a3f-serving-cert\""
Sep 09 03:32:21.208100 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.208077    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"]
Sep 09 03:32:21.231488 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.231459    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:21.268808 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.268782    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-maas-api-tls\") pod \"maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb\" (UID: \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:32:21.268943 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.268822    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-metrics-certs\") pod \"maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb\" (UID: \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:32:21.268943 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.268907    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-openshift-service-ca\") pod \"maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb\" (UID: \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:32:21.269029 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.268952    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnxrb\" (UniqueName: \"kubernetes.io/projected/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-kube-api-access-xnxrb\") pod \"maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb\" (UID: \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:32:21.307955 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.307923    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:21.369817 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.369780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-maas-api-tls\") pod \"maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb\" (UID: \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:32:21.369974 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.369847    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-metrics-certs\") pod \"maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb\" (UID: \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:32:21.369974 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:32:21.369943    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-serving-cert: secret "maas-api-e2e-explicit-958a3f-serving-cert" not found
Sep 09 03:32:21.369974 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.369951    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-openshift-service-ca\") pod \"maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb\" (UID: \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:32:21.370132 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.370003    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnxrb\" (UniqueName: \"kubernetes.io/projected/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-kube-api-access-xnxrb\") pod \"maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb\" (UID: \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:32:21.370132 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:32:21.370021    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-maas-api-tls podName:0f5698df-5555-4d0f-9d7b-bf1d856a0e99 nodeName:}" failed. No retries permitted until 2026-09-09 03:32:21.870002069 +0000 UTC m=+1159.006749935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-maas-api-tls") pod "maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb" (UID: "0f5698df-5555-4d0f-9d7b-bf1d856a0e99") : secret "maas-api-e2e-explicit-958a3f-serving-cert" not found
Sep 09 03:32:21.370654 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.370633    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-openshift-service-ca\") pod \"maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb\" (UID: \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:32:21.372115 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.372098    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-metrics-certs\") pod \"maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb\" (UID: \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:32:21.381539 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.381519    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnxrb\" (UniqueName: \"kubernetes.io/projected/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-kube-api-access-xnxrb\") pod \"maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb\" (UID: \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:32:21.508828 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.508721    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:21.745132 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.745090    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:21.836942 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.836854    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:21.875312 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.875275    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-maas-api-tls\") pod \"maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb\" (UID: \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:32:21.877558 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:21.877537    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-maas-api-tls\") pod \"maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb\" (UID: \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:32:22.113203 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:22.113165    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:32:22.386609 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:22.386559    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"]
Sep 09 03:32:22.387711 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:32:22.387679    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f5698df_5555_4d0f_9d7b_bf1d856a0e99.slice/crio-6c115d85bbf28bb0d7c09e3236434f8800f3bb588d15fb246217a4cdfbd8aed2 WatchSource:0}: Error finding container 6c115d85bbf28bb0d7c09e3236434f8800f3bb588d15fb246217a4cdfbd8aed2: Status 404 returned error can't find the container with id 6c115d85bbf28bb0d7c09e3236434f8800f3bb588d15fb246217a4cdfbd8aed2
Sep 09 03:32:22.802571 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:22.802485    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb" event={"ID":"0f5698df-5555-4d0f-9d7b-bf1d856a0e99","Type":"ContainerStarted","Data":"6c115d85bbf28bb0d7c09e3236434f8800f3bb588d15fb246217a4cdfbd8aed2"}
Sep 09 03:32:22.953487 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:22.953446    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm" podUID="abe67204-2ca8-499c-be1d-53fa2bc3e345" containerName="main" containerID="cri-o://7b5b01077db87a5e849cf207c4c1761cc5678e878e27598b97e06e4a9acaad94" gracePeriod=45
Sep 09 03:32:23.203952 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.203928    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:32:23.288293 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.288248    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-tmp-dir\") pod \"abe67204-2ca8-499c-be1d-53fa2bc3e345\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") "
Sep 09 03:32:23.288293 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.288299    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-home\") pod \"abe67204-2ca8-499c-be1d-53fa2bc3e345\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") "
Sep 09 03:32:23.288530 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.288374    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-model-cache\") pod \"abe67204-2ca8-499c-be1d-53fa2bc3e345\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") "
Sep 09 03:32:23.288530 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.288399    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe67204-2ca8-499c-be1d-53fa2bc3e345-tls-certs\") pod \"abe67204-2ca8-499c-be1d-53fa2bc3e345\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") "
Sep 09 03:32:23.288530 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.288451    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mm5nw\" (UniqueName: \"kubernetes.io/projected/abe67204-2ca8-499c-be1d-53fa2bc3e345-kube-api-access-mm5nw\") pod \"abe67204-2ca8-499c-be1d-53fa2bc3e345\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") "
Sep 09 03:32:23.288530 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.288508    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "abe67204-2ca8-499c-be1d-53fa2bc3e345" (UID: "abe67204-2ca8-499c-be1d-53fa2bc3e345"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:23.288782 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.288519    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-dshm\") pod \"abe67204-2ca8-499c-be1d-53fa2bc3e345\" (UID: \"abe67204-2ca8-499c-be1d-53fa2bc3e345\") "
Sep 09 03:32:23.288782 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.288632    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-home" (OuterVolumeSpecName: "home") pod "abe67204-2ca8-499c-be1d-53fa2bc3e345" (UID: "abe67204-2ca8-499c-be1d-53fa2bc3e345"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:23.288782 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.288663    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-model-cache" (OuterVolumeSpecName: "model-cache") pod "abe67204-2ca8-499c-be1d-53fa2bc3e345" (UID: "abe67204-2ca8-499c-be1d-53fa2bc3e345"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:23.288953 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.288927    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-tmp-dir\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:23.288953 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.288954    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-home\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:23.289107 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.288967    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-model-cache\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:23.290766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.290746    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-dshm" (OuterVolumeSpecName: "dshm") pod "abe67204-2ca8-499c-be1d-53fa2bc3e345" (UID: "abe67204-2ca8-499c-be1d-53fa2bc3e345"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:32:23.291065 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.291040    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe67204-2ca8-499c-be1d-53fa2bc3e345-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "abe67204-2ca8-499c-be1d-53fa2bc3e345" (UID: "abe67204-2ca8-499c-be1d-53fa2bc3e345"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:32:23.291065 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.291057    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe67204-2ca8-499c-be1d-53fa2bc3e345-kube-api-access-mm5nw" (OuterVolumeSpecName: "kube-api-access-mm5nw") pod "abe67204-2ca8-499c-be1d-53fa2bc3e345" (UID: "abe67204-2ca8-499c-be1d-53fa2bc3e345"). InnerVolumeSpecName "kube-api-access-mm5nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:32:23.389771 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.389747    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe67204-2ca8-499c-be1d-53fa2bc3e345-dshm\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:23.389771 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.389773    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe67204-2ca8-499c-be1d-53fa2bc3e345-tls-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:23.390119 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.389782    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm5nw\" (UniqueName: \"kubernetes.io/projected/abe67204-2ca8-499c-be1d-53fa2bc3e345-kube-api-access-mm5nw\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:32:23.808871 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.808773    2560 generic.go:356] "Generic (PLEG): container finished" podID="abe67204-2ca8-499c-be1d-53fa2bc3e345" containerID="7b5b01077db87a5e849cf207c4c1761cc5678e878e27598b97e06e4a9acaad94" exitCode=0
Sep 09 03:32:23.808871 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.808815    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"
Sep 09 03:32:23.808871 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.808858    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm" event={"ID":"abe67204-2ca8-499c-be1d-53fa2bc3e345","Type":"ContainerDied","Data":"7b5b01077db87a5e849cf207c4c1761cc5678e878e27598b97e06e4a9acaad94"}
Sep 09 03:32:23.809118 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.808900    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm" event={"ID":"abe67204-2ca8-499c-be1d-53fa2bc3e345","Type":"ContainerDied","Data":"cd3592fffda366410ccc5f3ea1790cfd70155c23f552e41f3b8a2bea5db39bcc"}
Sep 09 03:32:23.809118 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.808919    2560 scope.go:117] "RemoveContainer" containerID="7b5b01077db87a5e849cf207c4c1761cc5678e878e27598b97e06e4a9acaad94"
Sep 09 03:32:23.810944 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.810922    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb" event={"ID":"0f5698df-5555-4d0f-9d7b-bf1d856a0e99","Type":"ContainerStarted","Data":"b10cdf04f75c7de33336610d9bd0f288b05dab10aa5510b4a7b344fb87ac2f45"}
Sep 09 03:32:23.811101 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.811074    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:32:23.817811 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.817754    2560 scope.go:117] "RemoveContainer" containerID="7b5b01077db87a5e849cf207c4c1761cc5678e878e27598b97e06e4a9acaad94"
Sep 09 03:32:23.818161 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:32:23.818136    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5b01077db87a5e849cf207c4c1761cc5678e878e27598b97e06e4a9acaad94\": container with ID starting with 7b5b01077db87a5e849cf207c4c1761cc5678e878e27598b97e06e4a9acaad94 not found: ID does not exist" containerID="7b5b01077db87a5e849cf207c4c1761cc5678e878e27598b97e06e4a9acaad94"
Sep 09 03:32:23.818270 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.818169    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5b01077db87a5e849cf207c4c1761cc5678e878e27598b97e06e4a9acaad94"} err="failed to get container status \"7b5b01077db87a5e849cf207c4c1761cc5678e878e27598b97e06e4a9acaad94\": rpc error: code = NotFound desc = could not find container \"7b5b01077db87a5e849cf207c4c1761cc5678e878e27598b97e06e4a9acaad94\": container with ID starting with 7b5b01077db87a5e849cf207c4c1761cc5678e878e27598b97e06e4a9acaad94 not found: ID does not exist"
Sep 09 03:32:23.829034 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.829012    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"]
Sep 09 03:32:23.830895 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.830877    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-e893b71c-kserve-8c4cb85f9-7jqzm"]
Sep 09 03:32:23.852779 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:23.852737    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb" podStartSLOduration=2.356751997 podStartE2EDuration="2.852724157s" totalImagesPullingTime="495.97216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:32:23.85080231 +0000 UTC m=+1160.987550174" watchObservedRunningTime="2026-09-09 03:32:23.852724157 +0000 UTC m=+1160.989472022"
Sep 09 03:32:24.292086 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:24.292049    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:32:24.522606 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:24.522561    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:24.581332 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:24.581242    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:25.359104 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:25.359062    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abe67204-2ca8-499c-be1d-53fa2bc3e345" path="/var/lib/kubelet/pods/abe67204-2ca8-499c-be1d-53fa2bc3e345/volumes"
Sep 09 03:32:26.925905 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:26.925867    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:29.822214 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:29.822182    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:32:40.807832 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:40.807794    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:43.802529 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:43.802494    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:32:44.974617 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:44.974562    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:45.129470 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:45.129436    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:50.593174 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.593133    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"]
Sep 09 03:32:50.593579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.593531    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe67204-2ca8-499c-be1d-53fa2bc3e345" containerName="main"
Sep 09 03:32:50.593579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.593545    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe67204-2ca8-499c-be1d-53fa2bc3e345" containerName="main"
Sep 09 03:32:50.593579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.593555    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" containerName="istio-proxy"
Sep 09 03:32:50.593579 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.593561    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" containerName="istio-proxy"
Sep 09 03:32:50.593745 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.593673    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe67204-2ca8-499c-be1d-53fa2bc3e345" containerName="main"
Sep 09 03:32:50.593745 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.593689    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e25da1d-fa11-44f6-9c4b-38b38cfc8cb4" containerName="istio-proxy"
Sep 09 03:32:50.597181 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.597157    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.601321 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.601297    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-958a3f\"/\"kube-root-ca.crt\""
Sep 09 03:32:50.601456 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.601297    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-958a3f\"/\"explicit-model-958a3f-kserve-self-signed-certs\""
Sep 09 03:32:50.601456 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.601300    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-958a3f\"/\"openshift-service-ca.crt\""
Sep 09 03:32:50.601456 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.601347    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-958a3f\"/\"default-dockercfg-fslk6\""
Sep 09 03:32:50.612578 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.612552    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"]
Sep 09 03:32:50.655310 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.655276    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-home\") pod \"explicit-model-958a3f-kserve-686695777c-r9grw\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") " pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.655503 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.655387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-model-cache\") pod \"explicit-model-958a3f-kserve-686695777c-r9grw\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") " pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.655503 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.655460    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgkgj\" (UniqueName: \"kubernetes.io/projected/5a5d9352-8ad3-477d-b087-179d018718b9-kube-api-access-jgkgj\") pod \"explicit-model-958a3f-kserve-686695777c-r9grw\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") " pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.655503 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.655491    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-tmp-dir\") pod \"explicit-model-958a3f-kserve-686695777c-r9grw\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") " pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.655700 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.655525    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5d9352-8ad3-477d-b087-179d018718b9-tls-certs\") pod \"explicit-model-958a3f-kserve-686695777c-r9grw\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") " pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.655700 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.655556    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-dshm\") pod \"explicit-model-958a3f-kserve-686695777c-r9grw\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") " pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.692558 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.692527    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:50.756052 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.756022    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-model-cache\") pod \"explicit-model-958a3f-kserve-686695777c-r9grw\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") " pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.756216 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.756067    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgkgj\" (UniqueName: \"kubernetes.io/projected/5a5d9352-8ad3-477d-b087-179d018718b9-kube-api-access-jgkgj\") pod \"explicit-model-958a3f-kserve-686695777c-r9grw\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") " pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.756216 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.756087    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-tmp-dir\") pod \"explicit-model-958a3f-kserve-686695777c-r9grw\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") " pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.756216 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.756110    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5d9352-8ad3-477d-b087-179d018718b9-tls-certs\") pod \"explicit-model-958a3f-kserve-686695777c-r9grw\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") " pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.756216 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.756137    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-dshm\") pod \"explicit-model-958a3f-kserve-686695777c-r9grw\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") " pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.756216 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.756174    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-home\") pod \"explicit-model-958a3f-kserve-686695777c-r9grw\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") " pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.756485 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.756425    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-model-cache\") pod \"explicit-model-958a3f-kserve-686695777c-r9grw\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") " pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.756485 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.756471    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-tmp-dir\") pod \"explicit-model-958a3f-kserve-686695777c-r9grw\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") " pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.756589 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.756507    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-home\") pod \"explicit-model-958a3f-kserve-686695777c-r9grw\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") " pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.758533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.758510    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-dshm\") pod \"explicit-model-958a3f-kserve-686695777c-r9grw\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") " pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.758752 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.758737    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5d9352-8ad3-477d-b087-179d018718b9-tls-certs\") pod \"explicit-model-958a3f-kserve-686695777c-r9grw\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") " pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.764479 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.764455    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgkgj\" (UniqueName: \"kubernetes.io/projected/5a5d9352-8ad3-477d-b087-179d018718b9-kube-api-access-jgkgj\") pod \"explicit-model-958a3f-kserve-686695777c-r9grw\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") " pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:50.906240 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:50.906206    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:32:51.180019 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:51.179986    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"]
Sep 09 03:32:51.183704 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:32:51.183671    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a5d9352_8ad3_477d_b087_179d018718b9.slice/crio-b85964fea9212affd3d3ca2b3a9351f703f6cedd64133e9d9d04e483ef4b2a58 WatchSource:0}: Error finding container b85964fea9212affd3d3ca2b3a9351f703f6cedd64133e9d9d04e483ef4b2a58: Status 404 returned error can't find the container with id b85964fea9212affd3d3ca2b3a9351f703f6cedd64133e9d9d04e483ef4b2a58
Sep 09 03:32:51.385648 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:51.385610    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:51.492025 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:51.491943    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:51.697397 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:51.697365    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:51.955657 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:51.955619    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw" event={"ID":"5a5d9352-8ad3-477d-b087-179d018718b9","Type":"ContainerStarted","Data":"698c624f186cf4e7dab9f5edfb46ab341c3e68716ceec2c642e5c8df04c64e8e"}
Sep 09 03:32:51.955831 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:51.955668    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw" event={"ID":"5a5d9352-8ad3-477d-b087-179d018718b9","Type":"ContainerStarted","Data":"b85964fea9212affd3d3ca2b3a9351f703f6cedd64133e9d9d04e483ef4b2a58"}
Sep 09 03:32:51.976344 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:51.976298    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw" podStartSLOduration=1.976283663 podStartE2EDuration="1.976283663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:32:51.974485147 +0000 UTC m=+1189.111233012" watchObservedRunningTime="2026-09-09 03:32:51.976283663 +0000 UTC m=+1189.113031552"
Sep 09 03:32:52.947424 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:52.947387    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"]
Sep 09 03:32:52.977042 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:32:52.977008    2560 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-self-signed-certs: secret "explicit-model-958a3f-kserve-self-signed-certs" not found
Sep 09 03:32:52.977192 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:32:52.977090    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a5d9352-8ad3-477d-b087-179d018718b9-tls-certs podName:5a5d9352-8ad3-477d-b087-179d018718b9 nodeName:}" failed. No retries permitted until 2026-09-09 03:32:53.477068713 +0000 UTC m=+1190.613816569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5a5d9352-8ad3-477d-b087-179d018718b9-tls-certs") pod "explicit-model-958a3f-kserve-686695777c-r9grw" (UID: "5a5d9352-8ad3-477d-b087-179d018718b9") : secret "explicit-model-958a3f-kserve-self-signed-certs" not found
Sep 09 03:32:52.985868 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:52.985830    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:32:53.482881 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:32:53.482838    2560 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-self-signed-certs: secret "explicit-model-958a3f-kserve-self-signed-certs" not found
Sep 09 03:32:53.483091 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:32:53.482931    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a5d9352-8ad3-477d-b087-179d018718b9-tls-certs podName:5a5d9352-8ad3-477d-b087-179d018718b9 nodeName:}" failed. No retries permitted until 2026-09-09 03:32:54.482910789 +0000 UTC m=+1191.619658637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5a5d9352-8ad3-477d-b087-179d018718b9-tls-certs") pod "explicit-model-958a3f-kserve-686695777c-r9grw" (UID: "5a5d9352-8ad3-477d-b087-179d018718b9") : secret "explicit-model-958a3f-kserve-self-signed-certs" not found
Sep 09 03:32:54.492112 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:32:54.492075    2560 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-self-signed-certs: secret "explicit-model-958a3f-kserve-self-signed-certs" not found
Sep 09 03:32:54.492508 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:32:54.492154    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a5d9352-8ad3-477d-b087-179d018718b9-tls-certs podName:5a5d9352-8ad3-477d-b087-179d018718b9 nodeName:}" failed. No retries permitted until 2026-09-09 03:32:56.492139924 +0000 UTC m=+1193.628887766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5a5d9352-8ad3-477d-b087-179d018718b9-tls-certs") pod "explicit-model-958a3f-kserve-686695777c-r9grw" (UID: "5a5d9352-8ad3-477d-b087-179d018718b9") : secret "explicit-model-958a3f-kserve-self-signed-certs" not found
Sep 09 03:32:56.512029 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:32:56.511994    2560 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-self-signed-certs: secret "explicit-model-958a3f-kserve-self-signed-certs" not found
Sep 09 03:32:56.512415 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:32:56.512060    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a5d9352-8ad3-477d-b087-179d018718b9-tls-certs podName:5a5d9352-8ad3-477d-b087-179d018718b9 nodeName:}" failed. No retries permitted until 2026-09-09 03:33:00.512046638 +0000 UTC m=+1197.648794481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5a5d9352-8ad3-477d-b087-179d018718b9-tls-certs") pod "explicit-model-958a3f-kserve-686695777c-r9grw" (UID: "5a5d9352-8ad3-477d-b087-179d018718b9") : secret "explicit-model-958a3f-kserve-self-signed-certs" not found
Sep 09 03:32:58.198948 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:32:58.198911    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:33:00.551410 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:33:00.551369    2560 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-self-signed-certs: secret "explicit-model-958a3f-kserve-self-signed-certs" not found
Sep 09 03:33:00.551820 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:33:00.551457    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a5d9352-8ad3-477d-b087-179d018718b9-tls-certs podName:5a5d9352-8ad3-477d-b087-179d018718b9 nodeName:}" failed. No retries permitted until 2026-09-09 03:33:08.551440171 +0000 UTC m=+1205.688188014 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5a5d9352-8ad3-477d-b087-179d018718b9-tls-certs") pod "explicit-model-958a3f-kserve-686695777c-r9grw" (UID: "5a5d9352-8ad3-477d-b087-179d018718b9") : secret "explicit-model-958a3f-kserve-self-signed-certs" not found
Sep 09 03:33:00.906724 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:00.906691    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:33:03.662524 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:03.662490    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"]
Sep 09 03:33:03.662987 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:03.662790    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb" podUID="0f5698df-5555-4d0f-9d7b-bf1d856a0e99" containerName="maas-api" containerID="cri-o://b10cdf04f75c7de33336610d9bd0f288b05dab10aa5510b4a7b344fb87ac2f45" gracePeriod=30
Sep 09 03:33:03.700384 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:03.700348    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:03.799207 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:03.799172    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:03.901113 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:03.901074    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:03.911716 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:03.911694    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:33:03.982213 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:03.982182    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:03.986663 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:03.986638    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-openshift-service-ca\") pod \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\" (UID: \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\") "
Sep 09 03:33:03.986809 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:03.986682    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnxrb\" (UniqueName: \"kubernetes.io/projected/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-kube-api-access-xnxrb\") pod \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\" (UID: \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\") "
Sep 09 03:33:03.986809 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:03.986713    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-maas-api-tls\") pod \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\" (UID: \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\") "
Sep 09 03:33:03.986809 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:03.986759    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-metrics-certs\") pod \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\" (UID: \"0f5698df-5555-4d0f-9d7b-bf1d856a0e99\") "
Sep 09 03:33:03.987069 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:03.987045    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f5698df-5555-4d0f-9d7b-bf1d856a0e99" (UID: "0f5698df-5555-4d0f-9d7b-bf1d856a0e99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:33:03.987155 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:03.987112    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:03.988962 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:03.988925    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0f5698df-5555-4d0f-9d7b-bf1d856a0e99" (UID: "0f5698df-5555-4d0f-9d7b-bf1d856a0e99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:33:03.989072 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:03.988961    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f5698df-5555-4d0f-9d7b-bf1d856a0e99" (UID: "0f5698df-5555-4d0f-9d7b-bf1d856a0e99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:33:03.989072 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:03.989023    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-kube-api-access-xnxrb" (OuterVolumeSpecName: "kube-api-access-xnxrb") pod "0f5698df-5555-4d0f-9d7b-bf1d856a0e99" (UID: "0f5698df-5555-4d0f-9d7b-bf1d856a0e99"). InnerVolumeSpecName "kube-api-access-xnxrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:33:04.018070 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:04.018033    2560 generic.go:356] "Generic (PLEG): container finished" podID="0f5698df-5555-4d0f-9d7b-bf1d856a0e99" containerID="b10cdf04f75c7de33336610d9bd0f288b05dab10aa5510b4a7b344fb87ac2f45" exitCode=0
Sep 09 03:33:04.018245 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:04.018077    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"
Sep 09 03:33:04.018245 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:04.018106    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb" event={"ID":"0f5698df-5555-4d0f-9d7b-bf1d856a0e99","Type":"ContainerDied","Data":"b10cdf04f75c7de33336610d9bd0f288b05dab10aa5510b4a7b344fb87ac2f45"}
Sep 09 03:33:04.018245 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:04.018154    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb" event={"ID":"0f5698df-5555-4d0f-9d7b-bf1d856a0e99","Type":"ContainerDied","Data":"6c115d85bbf28bb0d7c09e3236434f8800f3bb588d15fb246217a4cdfbd8aed2"}
Sep 09 03:33:04.018245 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:04.018177    2560 scope.go:117] "RemoveContainer" containerID="b10cdf04f75c7de33336610d9bd0f288b05dab10aa5510b4a7b344fb87ac2f45"
Sep 09 03:33:04.033261 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:04.033240    2560 scope.go:117] "RemoveContainer" containerID="b10cdf04f75c7de33336610d9bd0f288b05dab10aa5510b4a7b344fb87ac2f45"
Sep 09 03:33:04.033602 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:33:04.033569    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10cdf04f75c7de33336610d9bd0f288b05dab10aa5510b4a7b344fb87ac2f45\": container with ID starting with b10cdf04f75c7de33336610d9bd0f288b05dab10aa5510b4a7b344fb87ac2f45 not found: ID does not exist" containerID="b10cdf04f75c7de33336610d9bd0f288b05dab10aa5510b4a7b344fb87ac2f45"
Sep 09 03:33:04.033665 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:04.033614    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10cdf04f75c7de33336610d9bd0f288b05dab10aa5510b4a7b344fb87ac2f45"} err="failed to get container status \"b10cdf04f75c7de33336610d9bd0f288b05dab10aa5510b4a7b344fb87ac2f45\": rpc error: code = NotFound desc = could not find container \"b10cdf04f75c7de33336610d9bd0f288b05dab10aa5510b4a7b344fb87ac2f45\": container with ID starting with b10cdf04f75c7de33336610d9bd0f288b05dab10aa5510b4a7b344fb87ac2f45 not found: ID does not exist"
Sep 09 03:33:04.066451 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:04.066415    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"]
Sep 09 03:33:04.069129 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:04.069103    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-958a3f-75f6ccfcb9-2jhgb"]
Sep 09 03:33:04.088690 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:04.088657    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnxrb\" (UniqueName: \"kubernetes.io/projected/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-kube-api-access-xnxrb\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:04.088690 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:04.088689    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:04.088868 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:04.088701    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5698df-5555-4d0f-9d7b-bf1d856a0e99-metrics-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:04.370194 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:04.370134    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:05.358508 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:05.358470    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f5698df-5555-4d0f-9d7b-bf1d856a0e99" path="/var/lib/kubelet/pods/0f5698df-5555-4d0f-9d7b-bf1d856a0e99/volumes"
Sep 09 03:33:07.524045 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:07.524008    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:08.633824 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:33:08.633790    2560 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-self-signed-certs: secret "explicit-model-958a3f-kserve-self-signed-certs" not found
Sep 09 03:33:08.634201 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:33:08.633866    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a5d9352-8ad3-477d-b087-179d018718b9-tls-certs podName:5a5d9352-8ad3-477d-b087-179d018718b9 nodeName:}" failed. No retries permitted until 2026-09-09 03:33:24.633850349 +0000 UTC m=+1221.770598208 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5a5d9352-8ad3-477d-b087-179d018718b9-tls-certs") pod "explicit-model-958a3f-kserve-686695777c-r9grw" (UID: "5a5d9352-8ad3-477d-b087-179d018718b9") : secret "explicit-model-958a3f-kserve-self-signed-certs" not found
Sep 09 03:33:08.985197 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:08.985101    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw" podUID="5a5d9352-8ad3-477d-b087-179d018718b9" containerName="main" containerID="cri-o://698c624f186cf4e7dab9f5edfb46ab341c3e68716ceec2c642e5c8df04c64e8e" gracePeriod=45
Sep 09 03:33:09.235862 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.235806    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:33:09.341024 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.340995    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-home\") pod \"5a5d9352-8ad3-477d-b087-179d018718b9\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") "
Sep 09 03:33:09.341170 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.341057    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-model-cache\") pod \"5a5d9352-8ad3-477d-b087-179d018718b9\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") "
Sep 09 03:33:09.341170 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.341077    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgkgj\" (UniqueName: \"kubernetes.io/projected/5a5d9352-8ad3-477d-b087-179d018718b9-kube-api-access-jgkgj\") pod \"5a5d9352-8ad3-477d-b087-179d018718b9\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") "
Sep 09 03:33:09.341170 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.341111    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5d9352-8ad3-477d-b087-179d018718b9-tls-certs\") pod \"5a5d9352-8ad3-477d-b087-179d018718b9\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") "
Sep 09 03:33:09.341170 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.341137    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-dshm\") pod \"5a5d9352-8ad3-477d-b087-179d018718b9\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") "
Sep 09 03:33:09.341395 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.341175    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-tmp-dir\") pod \"5a5d9352-8ad3-477d-b087-179d018718b9\" (UID: \"5a5d9352-8ad3-477d-b087-179d018718b9\") "
Sep 09 03:33:09.341395 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.341302    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a5d9352-8ad3-477d-b087-179d018718b9" (UID: "5a5d9352-8ad3-477d-b087-179d018718b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:09.341395 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.341315    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-home" (OuterVolumeSpecName: "home") pod "5a5d9352-8ad3-477d-b087-179d018718b9" (UID: "5a5d9352-8ad3-477d-b087-179d018718b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:09.341543 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.341464    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a5d9352-8ad3-477d-b087-179d018718b9" (UID: "5a5d9352-8ad3-477d-b087-179d018718b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:09.341628 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.341543    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-tmp-dir\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:09.341628 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.341561    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-home\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:09.341628 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.341570    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-model-cache\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:09.343320 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.343301    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-dshm" (OuterVolumeSpecName: "dshm") pod "5a5d9352-8ad3-477d-b087-179d018718b9" (UID: "5a5d9352-8ad3-477d-b087-179d018718b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:09.343414 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.343343    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5d9352-8ad3-477d-b087-179d018718b9-kube-api-access-jgkgj" (OuterVolumeSpecName: "kube-api-access-jgkgj") pod "5a5d9352-8ad3-477d-b087-179d018718b9" (UID: "5a5d9352-8ad3-477d-b087-179d018718b9"). InnerVolumeSpecName "kube-api-access-jgkgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:33:09.343414 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.343404    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a5d9352-8ad3-477d-b087-179d018718b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a5d9352-8ad3-477d-b087-179d018718b9" (UID: "5a5d9352-8ad3-477d-b087-179d018718b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:33:09.438563 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.438540    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:33:09.442105 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.442084    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5d9352-8ad3-477d-b087-179d018718b9-tls-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:09.442105 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.442104    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5d9352-8ad3-477d-b087-179d018718b9-dshm\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:09.442306 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.442116    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgkgj\" (UniqueName: \"kubernetes.io/projected/5a5d9352-8ad3-477d-b087-179d018718b9-kube-api-access-jgkgj\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:09.449767 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.449726    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:09.464828 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:09.464811    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:33:10.049447 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:10.049409    2560 generic.go:356] "Generic (PLEG): container finished" podID="5a5d9352-8ad3-477d-b087-179d018718b9" containerID="698c624f186cf4e7dab9f5edfb46ab341c3e68716ceec2c642e5c8df04c64e8e" exitCode=0
Sep 09 03:33:10.049867 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:10.049470    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"
Sep 09 03:33:10.049867 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:10.049493    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw" event={"ID":"5a5d9352-8ad3-477d-b087-179d018718b9","Type":"ContainerDied","Data":"698c624f186cf4e7dab9f5edfb46ab341c3e68716ceec2c642e5c8df04c64e8e"}
Sep 09 03:33:10.049867 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:10.049541    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw" event={"ID":"5a5d9352-8ad3-477d-b087-179d018718b9","Type":"ContainerDied","Data":"b85964fea9212affd3d3ca2b3a9351f703f6cedd64133e9d9d04e483ef4b2a58"}
Sep 09 03:33:10.049867 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:10.049559    2560 scope.go:117] "RemoveContainer" containerID="698c624f186cf4e7dab9f5edfb46ab341c3e68716ceec2c642e5c8df04c64e8e"
Sep 09 03:33:10.058677 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:10.058656    2560 scope.go:117] "RemoveContainer" containerID="698c624f186cf4e7dab9f5edfb46ab341c3e68716ceec2c642e5c8df04c64e8e"
Sep 09 03:33:10.058955 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:33:10.058932    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698c624f186cf4e7dab9f5edfb46ab341c3e68716ceec2c642e5c8df04c64e8e\": container with ID starting with 698c624f186cf4e7dab9f5edfb46ab341c3e68716ceec2c642e5c8df04c64e8e not found: ID does not exist" containerID="698c624f186cf4e7dab9f5edfb46ab341c3e68716ceec2c642e5c8df04c64e8e"
Sep 09 03:33:10.059012 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:10.058965    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698c624f186cf4e7dab9f5edfb46ab341c3e68716ceec2c642e5c8df04c64e8e"} err="failed to get container status \"698c624f186cf4e7dab9f5edfb46ab341c3e68716ceec2c642e5c8df04c64e8e\": rpc error: code = NotFound desc = could not find container \"698c624f186cf4e7dab9f5edfb46ab341c3e68716ceec2c642e5c8df04c64e8e\": container with ID starting with 698c624f186cf4e7dab9f5edfb46ab341c3e68716ceec2c642e5c8df04c64e8e not found: ID does not exist"
Sep 09 03:33:10.070487 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:10.070463    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"]
Sep 09 03:33:10.076485 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:10.076461    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-958a3f/explicit-model-958a3f-kserve-686695777c-r9grw"]
Sep 09 03:33:11.358841 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:11.358810    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a5d9352-8ad3-477d-b087-179d018718b9" path="/var/lib/kubelet/pods/5a5d9352-8ad3-477d-b087-179d018718b9/volumes"
Sep 09 03:33:12.006850 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:12.006820    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:12.097810 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:12.097781    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:12.221853 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:12.221820    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:12.741142 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:12.741106    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:13.128528 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:13.128489    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:13.264019 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:13.263988    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:14.246304 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:14.246268    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:14.390697 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:14.390654    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:14.507062 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:14.506972    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:14.753035 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:14.752996    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:14.843236 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:14.843135    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:15.001424 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:15.001390    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:15.993186 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:15.993150    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:33:25.044698 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:25.044656    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:25.876554 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:25.876508    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:28.789030 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.789001    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"]
Sep 09 03:33:28.789462 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.789445    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a5d9352-8ad3-477d-b087-179d018718b9" containerName="main"
Sep 09 03:33:28.789515 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.789464    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a5d9352-8ad3-477d-b087-179d018718b9" containerName="main"
Sep 09 03:33:28.789515 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.789483    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f5698df-5555-4d0f-9d7b-bf1d856a0e99" containerName="maas-api"
Sep 09 03:33:28.789515 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.789490    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f5698df-5555-4d0f-9d7b-bf1d856a0e99" containerName="maas-api"
Sep 09 03:33:28.789643 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.789552    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f5698df-5555-4d0f-9d7b-bf1d856a0e99" containerName="maas-api"
Sep 09 03:33:28.789643 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.789563    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a5d9352-8ad3-477d-b087-179d018718b9" containerName="main"
Sep 09 03:33:28.792785 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.792769    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:28.797918 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.797884    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-550c82\"/\"noten-model-550c82-kserve-self-signed-certs\""
Sep 09 03:33:28.799031 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.799007    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-550c82\"/\"kube-root-ca.crt\""
Sep 09 03:33:28.799159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.799028    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-550c82\"/\"default-dockercfg-8vrxg\""
Sep 09 03:33:28.799159 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.799051    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-550c82\"/\"openshift-service-ca.crt\""
Sep 09 03:33:28.807294 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.807271    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"]
Sep 09 03:33:28.834549 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.834517    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:28.920681 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.920647    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12a21b3-bec3-4d43-a021-3ee92bbef36c-tls-certs\") pod \"noten-model-550c82-kserve-78545f4955-pwxwk\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") " pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:28.920681 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.920685    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-tmp-dir\") pod \"noten-model-550c82-kserve-78545f4955-pwxwk\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") " pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:28.920897 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.920710    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-model-cache\") pod \"noten-model-550c82-kserve-78545f4955-pwxwk\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") " pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:28.920897 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.920801    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x7cp\" (UniqueName: \"kubernetes.io/projected/b12a21b3-bec3-4d43-a021-3ee92bbef36c-kube-api-access-8x7cp\") pod \"noten-model-550c82-kserve-78545f4955-pwxwk\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") " pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:28.920975 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.920907    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-dshm\") pod \"noten-model-550c82-kserve-78545f4955-pwxwk\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") " pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:28.920975 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:28.920951    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-home\") pod \"noten-model-550c82-kserve-78545f4955-pwxwk\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") " pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:29.021979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:29.021935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x7cp\" (UniqueName: \"kubernetes.io/projected/b12a21b3-bec3-4d43-a021-3ee92bbef36c-kube-api-access-8x7cp\") pod \"noten-model-550c82-kserve-78545f4955-pwxwk\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") " pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:29.022130 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:29.022002    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-dshm\") pod \"noten-model-550c82-kserve-78545f4955-pwxwk\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") " pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:29.022130 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:29.022048    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-home\") pod \"noten-model-550c82-kserve-78545f4955-pwxwk\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") " pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:29.022130 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:29.022093    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12a21b3-bec3-4d43-a021-3ee92bbef36c-tls-certs\") pod \"noten-model-550c82-kserve-78545f4955-pwxwk\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") " pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:29.022130 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:29.022115    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-tmp-dir\") pod \"noten-model-550c82-kserve-78545f4955-pwxwk\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") " pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:29.022275 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:29.022153    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-model-cache\") pod \"noten-model-550c82-kserve-78545f4955-pwxwk\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") " pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:29.022502 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:29.022475    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-home\") pod \"noten-model-550c82-kserve-78545f4955-pwxwk\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") " pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:29.022502 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:29.022493    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-model-cache\") pod \"noten-model-550c82-kserve-78545f4955-pwxwk\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") " pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:29.022705 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:29.022556    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-tmp-dir\") pod \"noten-model-550c82-kserve-78545f4955-pwxwk\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") " pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:29.024321 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:29.024294    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-dshm\") pod \"noten-model-550c82-kserve-78545f4955-pwxwk\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") " pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:29.024458 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:29.024441    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12a21b3-bec3-4d43-a021-3ee92bbef36c-tls-certs\") pod \"noten-model-550c82-kserve-78545f4955-pwxwk\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") " pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:29.033126 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:29.033106    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x7cp\" (UniqueName: \"kubernetes.io/projected/b12a21b3-bec3-4d43-a021-3ee92bbef36c-kube-api-access-8x7cp\") pod \"noten-model-550c82-kserve-78545f4955-pwxwk\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") " pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:29.102841 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:29.102822    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:29.378717 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:29.378691    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"]
Sep 09 03:33:29.380611 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:33:29.380571    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb12a21b3_bec3_4d43_a021_3ee92bbef36c.slice/crio-a223062468ad7118e6fa9a38128ead638cb84a061f311f701e60c29d82b5a968 WatchSource:0}: Error finding container a223062468ad7118e6fa9a38128ead638cb84a061f311f701e60c29d82b5a968: Status 404 returned error can't find the container with id a223062468ad7118e6fa9a38128ead638cb84a061f311f701e60c29d82b5a968
Sep 09 03:33:30.153948 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:30.153909    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk" event={"ID":"b12a21b3-bec3-4d43-a021-3ee92bbef36c","Type":"ContainerStarted","Data":"3a442ee110aeedf59fe66567631b052c85a27ebb22ac83f2bc59426dfd566c60"}
Sep 09 03:33:30.153948 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:30.153954    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk" event={"ID":"b12a21b3-bec3-4d43-a021-3ee92bbef36c","Type":"ContainerStarted","Data":"a223062468ad7118e6fa9a38128ead638cb84a061f311f701e60c29d82b5a968"}
Sep 09 03:33:30.176134 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:30.176081    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk" podStartSLOduration=2.176062825 podStartE2EDuration="2.176062825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:33:30.172913425 +0000 UTC m=+1227.309661291" watchObservedRunningTime="2026-09-09 03:33:30.176062825 +0000 UTC m=+1227.312810709"
Sep 09 03:33:32.222836 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:32.222800    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:33:32.235945 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:32.235914    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:32.399398 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:32.399357    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:32.525383 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:32.525293    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:33.392124 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:33.392073    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:33.490887 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:33.490858    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:35.156329 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:35.156294    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:36.753296 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:36.753263    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"]
Sep 09 03:33:36.783453 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:36.783422    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:36.938428 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:36.938374    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:37.171142 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:37.171104    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:37.293936 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:37.293900    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:39.102939 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:39.102893    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:40.140623 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:40.140577    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"]
Sep 09 03:33:40.169860 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:40.169834    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:41.471758 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:41.471727    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:41.761147 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:41.761057    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:42.358684 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:42.358649    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:42.442366 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:42.442327    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:43.061406 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:43.061371    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:46.321974 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:46.321936    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:33:49.563500 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:49.563440    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:49.657508 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:49.657458    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:49.834153 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:49.834043    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:50.225935 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:50.225896    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:50.459984 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:50.459941    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:51.772822 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:51.772758    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk" podUID="b12a21b3-bec3-4d43-a021-3ee92bbef36c" containerName="main" containerID="cri-o://3a442ee110aeedf59fe66567631b052c85a27ebb22ac83f2bc59426dfd566c60" gracePeriod=45
Sep 09 03:33:52.012296 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.012273    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:52.125324 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.125289    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-model-cache\") pod \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") "
Sep 09 03:33:52.125514 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.125389    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8x7cp\" (UniqueName: \"kubernetes.io/projected/b12a21b3-bec3-4d43-a021-3ee92bbef36c-kube-api-access-8x7cp\") pod \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") "
Sep 09 03:33:52.125514 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.125426    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12a21b3-bec3-4d43-a021-3ee92bbef36c-tls-certs\") pod \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") "
Sep 09 03:33:52.125514 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.125464    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-dshm\") pod \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") "
Sep 09 03:33:52.125514 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.125497    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-tmp-dir\") pod \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") "
Sep 09 03:33:52.125749 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.125545    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-home\") pod \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\" (UID: \"b12a21b3-bec3-4d43-a021-3ee92bbef36c\") "
Sep 09 03:33:52.125749 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.125680    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-model-cache" (OuterVolumeSpecName: "model-cache") pod "b12a21b3-bec3-4d43-a021-3ee92bbef36c" (UID: "b12a21b3-bec3-4d43-a021-3ee92bbef36c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:52.126128 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.125927    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-model-cache\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:52.126128 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.126010    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b12a21b3-bec3-4d43-a021-3ee92bbef36c" (UID: "b12a21b3-bec3-4d43-a021-3ee92bbef36c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:52.126128 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.126089    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-home" (OuterVolumeSpecName: "home") pod "b12a21b3-bec3-4d43-a021-3ee92bbef36c" (UID: "b12a21b3-bec3-4d43-a021-3ee92bbef36c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:52.130028 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.129939    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12a21b3-bec3-4d43-a021-3ee92bbef36c-kube-api-access-8x7cp" (OuterVolumeSpecName: "kube-api-access-8x7cp") pod "b12a21b3-bec3-4d43-a021-3ee92bbef36c" (UID: "b12a21b3-bec3-4d43-a021-3ee92bbef36c"). InnerVolumeSpecName "kube-api-access-8x7cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:33:52.130155 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.130030    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-dshm" (OuterVolumeSpecName: "dshm") pod "b12a21b3-bec3-4d43-a021-3ee92bbef36c" (UID: "b12a21b3-bec3-4d43-a021-3ee92bbef36c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:52.130155 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.130060    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b12a21b3-bec3-4d43-a021-3ee92bbef36c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b12a21b3-bec3-4d43-a021-3ee92bbef36c" (UID: "b12a21b3-bec3-4d43-a021-3ee92bbef36c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:33:52.197791 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.197679    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:52.226678 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.226645    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x7cp\" (UniqueName: \"kubernetes.io/projected/b12a21b3-bec3-4d43-a021-3ee92bbef36c-kube-api-access-8x7cp\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:52.226678 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.226673    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12a21b3-bec3-4d43-a021-3ee92bbef36c-tls-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:52.226678 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.226684    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-dshm\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:52.226902 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.226696    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-tmp-dir\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:52.226902 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.226706    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b12a21b3-bec3-4d43-a021-3ee92bbef36c-home\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:52.269802 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.269763    2560 generic.go:356] "Generic (PLEG): container finished" podID="b12a21b3-bec3-4d43-a021-3ee92bbef36c" containerID="3a442ee110aeedf59fe66567631b052c85a27ebb22ac83f2bc59426dfd566c60" exitCode=0
Sep 09 03:33:52.269971 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.269808    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"
Sep 09 03:33:52.269971 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.269840    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk" event={"ID":"b12a21b3-bec3-4d43-a021-3ee92bbef36c","Type":"ContainerDied","Data":"3a442ee110aeedf59fe66567631b052c85a27ebb22ac83f2bc59426dfd566c60"}
Sep 09 03:33:52.269971 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.269899    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk" event={"ID":"b12a21b3-bec3-4d43-a021-3ee92bbef36c","Type":"ContainerDied","Data":"a223062468ad7118e6fa9a38128ead638cb84a061f311f701e60c29d82b5a968"}
Sep 09 03:33:52.269971 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.269926    2560 scope.go:117] "RemoveContainer" containerID="3a442ee110aeedf59fe66567631b052c85a27ebb22ac83f2bc59426dfd566c60"
Sep 09 03:33:52.281976 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.281953    2560 scope.go:117] "RemoveContainer" containerID="3a442ee110aeedf59fe66567631b052c85a27ebb22ac83f2bc59426dfd566c60"
Sep 09 03:33:52.282370 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:33:52.282349    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a442ee110aeedf59fe66567631b052c85a27ebb22ac83f2bc59426dfd566c60\": container with ID starting with 3a442ee110aeedf59fe66567631b052c85a27ebb22ac83f2bc59426dfd566c60 not found: ID does not exist" containerID="3a442ee110aeedf59fe66567631b052c85a27ebb22ac83f2bc59426dfd566c60"
Sep 09 03:33:52.282479 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.282383    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a442ee110aeedf59fe66567631b052c85a27ebb22ac83f2bc59426dfd566c60"} err="failed to get container status \"3a442ee110aeedf59fe66567631b052c85a27ebb22ac83f2bc59426dfd566c60\": rpc error: code = NotFound desc = could not find container \"3a442ee110aeedf59fe66567631b052c85a27ebb22ac83f2bc59426dfd566c60\": container with ID starting with 3a442ee110aeedf59fe66567631b052c85a27ebb22ac83f2bc59426dfd566c60 not found: ID does not exist"
Sep 09 03:33:52.303319 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.303293    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"]
Sep 09 03:33:52.307065 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.307041    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-550c82/noten-model-550c82-kserve-78545f4955-pwxwk"]
Sep 09 03:33:52.319703 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.319676    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:52.421051 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.420970    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:52.506050 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.506011    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:52.976185 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:52.976153    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:53.360182 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:53.360153    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b12a21b3-bec3-4d43-a021-3ee92bbef36c" path="/var/lib/kubelet/pods/b12a21b3-bec3-4d43-a021-3ee92bbef36c/volumes"
Sep 09 03:33:53.708177 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:53.708064    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:53.854356 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:53.854321    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:54.301885 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:54.301851    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:33:54.858001 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:54.857969    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"]
Sep 09 03:33:54.858424 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:54.858400    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b12a21b3-bec3-4d43-a021-3ee92bbef36c" containerName="main"
Sep 09 03:33:54.858515 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:54.858428    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b12a21b3-bec3-4d43-a021-3ee92bbef36c" containerName="main"
Sep 09 03:33:54.858589 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:54.858575    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b12a21b3-bec3-4d43-a021-3ee92bbef36c" containerName="main"
Sep 09 03:33:54.863073 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:54.863044    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:33:54.866057 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:54.866039    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c44cc65a-serving-cert\""
Sep 09 03:33:54.887021 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:54.886990    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"]
Sep 09 03:33:54.953647 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:54.953617    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92a8360-f72c-4bfd-a200-36e13fa4c168-metrics-certs\") pod \"maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz\" (UID: \"a92a8360-f72c-4bfd-a200-36e13fa4c168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:33:54.953820 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:54.953703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fqxb\" (UniqueName: \"kubernetes.io/projected/a92a8360-f72c-4bfd-a200-36e13fa4c168-kube-api-access-9fqxb\") pod \"maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz\" (UID: \"a92a8360-f72c-4bfd-a200-36e13fa4c168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:33:54.953820 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:54.953739    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92a8360-f72c-4bfd-a200-36e13fa4c168-openshift-service-ca\") pod \"maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz\" (UID: \"a92a8360-f72c-4bfd-a200-36e13fa4c168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:33:54.953820 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:54.953771    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a92a8360-f72c-4bfd-a200-36e13fa4c168-maas-api-tls\") pod \"maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz\" (UID: \"a92a8360-f72c-4bfd-a200-36e13fa4c168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:33:54.973171 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:54.973143    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:55.055030 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.054988    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92a8360-f72c-4bfd-a200-36e13fa4c168-openshift-service-ca\") pod \"maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz\" (UID: \"a92a8360-f72c-4bfd-a200-36e13fa4c168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:33:55.055437 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.055038    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a92a8360-f72c-4bfd-a200-36e13fa4c168-maas-api-tls\") pod \"maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz\" (UID: \"a92a8360-f72c-4bfd-a200-36e13fa4c168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:33:55.055437 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.055094    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92a8360-f72c-4bfd-a200-36e13fa4c168-metrics-certs\") pod \"maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz\" (UID: \"a92a8360-f72c-4bfd-a200-36e13fa4c168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:33:55.055437 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.055141    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fqxb\" (UniqueName: \"kubernetes.io/projected/a92a8360-f72c-4bfd-a200-36e13fa4c168-kube-api-access-9fqxb\") pod \"maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz\" (UID: \"a92a8360-f72c-4bfd-a200-36e13fa4c168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:33:55.055437 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:33:55.055207    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-serving-cert: secret "maas-api-e2e-mt-c44cc65a-serving-cert" not found
Sep 09 03:33:55.055437 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:33:55.055279    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92a8360-f72c-4bfd-a200-36e13fa4c168-maas-api-tls podName:a92a8360-f72c-4bfd-a200-36e13fa4c168 nodeName:}" failed. No retries permitted until 2026-09-09 03:33:55.555260912 +0000 UTC m=+1252.692008756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a92a8360-f72c-4bfd-a200-36e13fa4c168-maas-api-tls") pod "maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz" (UID: "a92a8360-f72c-4bfd-a200-36e13fa4c168") : secret "maas-api-e2e-mt-c44cc65a-serving-cert" not found
Sep 09 03:33:55.055751 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.055732    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92a8360-f72c-4bfd-a200-36e13fa4c168-openshift-service-ca\") pod \"maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz\" (UID: \"a92a8360-f72c-4bfd-a200-36e13fa4c168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:33:55.057350 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.057323    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92a8360-f72c-4bfd-a200-36e13fa4c168-metrics-certs\") pod \"maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz\" (UID: \"a92a8360-f72c-4bfd-a200-36e13fa4c168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:33:55.073851 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.073827    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fqxb\" (UniqueName: \"kubernetes.io/projected/a92a8360-f72c-4bfd-a200-36e13fa4c168-kube-api-access-9fqxb\") pod \"maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz\" (UID: \"a92a8360-f72c-4bfd-a200-36e13fa4c168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:33:55.097627 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.097587    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:55.159892 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.159798    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg" podUID="e8958171-2e22-4a18-bdc6-5df916b64058" containerName="main" containerID="cri-o://21b258b74932d2df96463d9248d9a41b97402582914e358f3bb64c1c1408cb3f" gracePeriod=45
Sep 09 03:33:55.221023 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.220974    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:55.289306 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.289277    2560 generic.go:356] "Generic (PLEG): container finished" podID="e8958171-2e22-4a18-bdc6-5df916b64058" containerID="21b258b74932d2df96463d9248d9a41b97402582914e358f3bb64c1c1408cb3f" exitCode=0
Sep 09 03:33:55.289452 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.289344    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg" event={"ID":"e8958171-2e22-4a18-bdc6-5df916b64058","Type":"ContainerDied","Data":"21b258b74932d2df96463d9248d9a41b97402582914e358f3bb64c1c1408cb3f"}
Sep 09 03:33:55.400803 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.400780    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:33:55.458126 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.458035    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-home\") pod \"e8958171-2e22-4a18-bdc6-5df916b64058\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") "
Sep 09 03:33:55.458126 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.458103    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-tmp-dir\") pod \"e8958171-2e22-4a18-bdc6-5df916b64058\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") "
Sep 09 03:33:55.458344 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.458144    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8958171-2e22-4a18-bdc6-5df916b64058-tls-certs\") pod \"e8958171-2e22-4a18-bdc6-5df916b64058\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") "
Sep 09 03:33:55.458344 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.458169    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-dshm\") pod \"e8958171-2e22-4a18-bdc6-5df916b64058\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") "
Sep 09 03:33:55.458344 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.458211    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-model-cache\") pod \"e8958171-2e22-4a18-bdc6-5df916b64058\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") "
Sep 09 03:33:55.458344 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.458268    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x27gz\" (UniqueName: \"kubernetes.io/projected/e8958171-2e22-4a18-bdc6-5df916b64058-kube-api-access-x27gz\") pod \"e8958171-2e22-4a18-bdc6-5df916b64058\" (UID: \"e8958171-2e22-4a18-bdc6-5df916b64058\") "
Sep 09 03:33:55.458558 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.458359    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8958171-2e22-4a18-bdc6-5df916b64058" (UID: "e8958171-2e22-4a18-bdc6-5df916b64058"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:55.458558 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.458369    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-home" (OuterVolumeSpecName: "home") pod "e8958171-2e22-4a18-bdc6-5df916b64058" (UID: "e8958171-2e22-4a18-bdc6-5df916b64058"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:55.458694 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.458645    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-home\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:55.458694 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.458664    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-tmp-dir\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:55.458835 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.458806    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8958171-2e22-4a18-bdc6-5df916b64058" (UID: "e8958171-2e22-4a18-bdc6-5df916b64058"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:55.460561 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.460533    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-dshm" (OuterVolumeSpecName: "dshm") pod "e8958171-2e22-4a18-bdc6-5df916b64058" (UID: "e8958171-2e22-4a18-bdc6-5df916b64058"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:33:55.460881 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.460862    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8958171-2e22-4a18-bdc6-5df916b64058-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8958171-2e22-4a18-bdc6-5df916b64058" (UID: "e8958171-2e22-4a18-bdc6-5df916b64058"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:33:55.460950 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.460894    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8958171-2e22-4a18-bdc6-5df916b64058-kube-api-access-x27gz" (OuterVolumeSpecName: "kube-api-access-x27gz") pod "e8958171-2e22-4a18-bdc6-5df916b64058" (UID: "e8958171-2e22-4a18-bdc6-5df916b64058"). InnerVolumeSpecName "kube-api-access-x27gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:33:55.559152 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.559116    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a92a8360-f72c-4bfd-a200-36e13fa4c168-maas-api-tls\") pod \"maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz\" (UID: \"a92a8360-f72c-4bfd-a200-36e13fa4c168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:33:55.559408 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.559266    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8958171-2e22-4a18-bdc6-5df916b64058-tls-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:55.559408 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.559282    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-dshm\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:55.559408 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.559295    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8958171-2e22-4a18-bdc6-5df916b64058-model-cache\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:55.559408 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.559307    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x27gz\" (UniqueName: \"kubernetes.io/projected/e8958171-2e22-4a18-bdc6-5df916b64058-kube-api-access-x27gz\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:33:55.561489 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.561470    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a92a8360-f72c-4bfd-a200-36e13fa4c168-maas-api-tls\") pod \"maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz\" (UID: \"a92a8360-f72c-4bfd-a200-36e13fa4c168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:33:55.729011 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.728883    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:55.776368 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.776338    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:33:55.861581 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:55.861551    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:33:56.054392 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:56.054365    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"]
Sep 09 03:33:56.056621 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:33:56.056570    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92a8360_f72c_4bfd_a200_36e13fa4c168.slice/crio-39201b1331fb19764e32122c587753e5c1b89b097b17daf6a803ef3ecc2362d6 WatchSource:0}: Error finding container 39201b1331fb19764e32122c587753e5c1b89b097b17daf6a803ef3ecc2362d6: Status 404 returned error can't find the container with id 39201b1331fb19764e32122c587753e5c1b89b097b17daf6a803ef3ecc2362d6
Sep 09 03:33:56.058042 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:56.058022    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:33:56.296326 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:56.296293    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"
Sep 09 03:33:56.296512 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:56.296300    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg" event={"ID":"e8958171-2e22-4a18-bdc6-5df916b64058","Type":"ContainerDied","Data":"afa84eac0c0d959078d9142b3cfbd5ddbaf7e0893822a22b32fa887810e7c361"}
Sep 09 03:33:56.296512 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:56.296415    2560 scope.go:117] "RemoveContainer" containerID="21b258b74932d2df96463d9248d9a41b97402582914e358f3bb64c1c1408cb3f"
Sep 09 03:33:56.297809 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:56.297470    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz" event={"ID":"a92a8360-f72c-4bfd-a200-36e13fa4c168","Type":"ContainerStarted","Data":"39201b1331fb19764e32122c587753e5c1b89b097b17daf6a803ef3ecc2362d6"}
Sep 09 03:33:56.318373 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:56.318343    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"]
Sep 09 03:33:56.325548 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:56.325524    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-40a1c9/test-model-40a1c9-kserve-79cdb4454f-bj9kg"]
Sep 09 03:33:57.305804 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:57.305769    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz" event={"ID":"a92a8360-f72c-4bfd-a200-36e13fa4c168","Type":"ContainerStarted","Data":"0cc7e64b398b3985a9444444cf9076d664d129ae1bb3f986581ceec4bfa5a5b5"}
Sep 09 03:33:57.306182 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:57.305887    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:33:57.334340 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:57.334283    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz" podStartSLOduration=2.927758043 podStartE2EDuration="3.334263294s" totalImagesPullingTime="406.505251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:33:57.333463659 +0000 UTC m=+1254.470211524" watchObservedRunningTime="2026-09-09 03:33:57.334263294 +0000 UTC m=+1254.471011160"
Sep 09 03:33:57.359062 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:57.359031    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8958171-2e22-4a18-bdc6-5df916b64058" path="/var/lib/kubelet/pods/e8958171-2e22-4a18-bdc6-5df916b64058/volumes"
Sep 09 03:33:59.121470 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:33:59.121433    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:00.617050 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:00.617020    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:03.100446 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:03.100409    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:34:03.317119 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:03.317091    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:34:10.717535 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:10.717502    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"]
Sep 09 03:34:10.717965 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:10.717735    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs" podUID="e82c204d-cb79-481e-ade0-255db3330688" containerName="maas-api" containerID="cri-o://5e2c6993478ab9e08572f8533278412003f9f34013c8afaa95d66faf524d4700" gracePeriod=30
Sep 09 03:34:10.814769 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:10.814734    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:10.965634 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:10.965581    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:34:11.007842 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.007770    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:11.104562 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.104534    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6hlh\" (UniqueName: \"kubernetes.io/projected/e82c204d-cb79-481e-ade0-255db3330688-kube-api-access-z6hlh\") pod \"e82c204d-cb79-481e-ade0-255db3330688\" (UID: \"e82c204d-cb79-481e-ade0-255db3330688\") "
Sep 09 03:34:11.104734 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.104587    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82c204d-cb79-481e-ade0-255db3330688-openshift-service-ca\") pod \"e82c204d-cb79-481e-ade0-255db3330688\" (UID: \"e82c204d-cb79-481e-ade0-255db3330688\") "
Sep 09 03:34:11.104734 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.104673    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e82c204d-cb79-481e-ade0-255db3330688-metrics-certs\") pod \"e82c204d-cb79-481e-ade0-255db3330688\" (UID: \"e82c204d-cb79-481e-ade0-255db3330688\") "
Sep 09 03:34:11.104852 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.104737    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82c204d-cb79-481e-ade0-255db3330688-maas-api-tls\") pod \"e82c204d-cb79-481e-ade0-255db3330688\" (UID: \"e82c204d-cb79-481e-ade0-255db3330688\") "
Sep 09 03:34:11.104975 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.104952    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e82c204d-cb79-481e-ade0-255db3330688-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e82c204d-cb79-481e-ade0-255db3330688" (UID: "e82c204d-cb79-481e-ade0-255db3330688"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:34:11.106750 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.106717    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e82c204d-cb79-481e-ade0-255db3330688-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e82c204d-cb79-481e-ade0-255db3330688" (UID: "e82c204d-cb79-481e-ade0-255db3330688"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:34:11.106868 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.106824    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82c204d-cb79-481e-ade0-255db3330688-kube-api-access-z6hlh" (OuterVolumeSpecName: "kube-api-access-z6hlh") pod "e82c204d-cb79-481e-ade0-255db3330688" (UID: "e82c204d-cb79-481e-ade0-255db3330688"). InnerVolumeSpecName "kube-api-access-z6hlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:34:11.106868 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.106831    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e82c204d-cb79-481e-ade0-255db3330688-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e82c204d-cb79-481e-ade0-255db3330688" (UID: "e82c204d-cb79-481e-ade0-255db3330688"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:34:11.199610 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.199562    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:11.206326 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.206304    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e82c204d-cb79-481e-ade0-255db3330688-metrics-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:34:11.206326 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.206325    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82c204d-cb79-481e-ade0-255db3330688-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:34:11.206463 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.206334    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6hlh\" (UniqueName: \"kubernetes.io/projected/e82c204d-cb79-481e-ade0-255db3330688-kube-api-access-z6hlh\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:34:11.206463 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.206351    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82c204d-cb79-481e-ade0-255db3330688-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:34:11.377143 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.377112    2560 generic.go:356] "Generic (PLEG): container finished" podID="e82c204d-cb79-481e-ade0-255db3330688" containerID="5e2c6993478ab9e08572f8533278412003f9f34013c8afaa95d66faf524d4700" exitCode=0
Sep 09 03:34:11.377286 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.377166    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"
Sep 09 03:34:11.377286 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.377192    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs" event={"ID":"e82c204d-cb79-481e-ade0-255db3330688","Type":"ContainerDied","Data":"5e2c6993478ab9e08572f8533278412003f9f34013c8afaa95d66faf524d4700"}
Sep 09 03:34:11.377286 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.377245    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs" event={"ID":"e82c204d-cb79-481e-ade0-255db3330688","Type":"ContainerDied","Data":"2fcffe4e6390f1a2c10b3318148b4cedb478ca8ce5b0d3b717ecbc2f1de80972"}
Sep 09 03:34:11.377286 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.377266    2560 scope.go:117] "RemoveContainer" containerID="5e2c6993478ab9e08572f8533278412003f9f34013c8afaa95d66faf524d4700"
Sep 09 03:34:11.385522 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.385501    2560 scope.go:117] "RemoveContainer" containerID="5e2c6993478ab9e08572f8533278412003f9f34013c8afaa95d66faf524d4700"
Sep 09 03:34:11.385870 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:34:11.385849    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2c6993478ab9e08572f8533278412003f9f34013c8afaa95d66faf524d4700\": container with ID starting with 5e2c6993478ab9e08572f8533278412003f9f34013c8afaa95d66faf524d4700 not found: ID does not exist" containerID="5e2c6993478ab9e08572f8533278412003f9f34013c8afaa95d66faf524d4700"
Sep 09 03:34:11.385916 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.385882    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2c6993478ab9e08572f8533278412003f9f34013c8afaa95d66faf524d4700"} err="failed to get container status \"5e2c6993478ab9e08572f8533278412003f9f34013c8afaa95d66faf524d4700\": rpc error: code = NotFound desc = could not find container \"5e2c6993478ab9e08572f8533278412003f9f34013c8afaa95d66faf524d4700\": container with ID starting with 5e2c6993478ab9e08572f8533278412003f9f34013c8afaa95d66faf524d4700 not found: ID does not exist"
Sep 09 03:34:11.418604 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.418546    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:11.421831 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.421798    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"]
Sep 09 03:34:11.427469 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.427445    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4bd6b8-58f889cb75-p8fjs"]
Sep 09 03:34:11.531059 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.531026    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:11.813178 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:11.813087    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:13.358962 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:13.358928    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e82c204d-cb79-481e-ade0-255db3330688" path="/var/lib/kubelet/pods/e82c204d-cb79-481e-ade0-255db3330688/volumes"
Sep 09 03:34:13.696420 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:13.696342    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:34:16.240785 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:16.240748    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:18.455941 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:18.455899    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:18.566691 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:18.566657    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:18.661076 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:18.661039    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:18.797493 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:18.797412    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:19.464387 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:19.464354    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:19.616784 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:19.616753    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:19.880719 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:19.880687    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:20.006569 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:20.006539    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:20.151446 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:20.151350    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:21.256061 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:21.256025    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:23.096143 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:23.096112    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:23.182533 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:23.182503    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:24.296055 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.296015    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"]
Sep 09 03:34:24.296637 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.296620    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8958171-2e22-4a18-bdc6-5df916b64058" containerName="main"
Sep 09 03:34:24.296705 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.296639    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8958171-2e22-4a18-bdc6-5df916b64058" containerName="main"
Sep 09 03:34:24.296705 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.296657    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e82c204d-cb79-481e-ade0-255db3330688" containerName="maas-api"
Sep 09 03:34:24.296705 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.296666    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e82c204d-cb79-481e-ade0-255db3330688" containerName="maas-api"
Sep 09 03:34:24.296815 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.296755    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8958171-2e22-4a18-bdc6-5df916b64058" containerName="main"
Sep 09 03:34:24.296815 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.296771    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e82c204d-cb79-481e-ade0-255db3330688" containerName="maas-api"
Sep 09 03:34:24.301656 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.301638    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:34:24.304664 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.304642    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-5d8e85-serving-cert\""
Sep 09 03:34:24.321849 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.321824    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"]
Sep 09 03:34:24.325401 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.325372    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx257\" (UniqueName: \"kubernetes.io/projected/b4ea9840-b602-4676-a8f2-122a5403a336-kube-api-access-nx257\") pod \"maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm\" (UID: \"b4ea9840-b602-4676-a8f2-122a5403a336\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:34:24.325491 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.325424    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea9840-b602-4676-a8f2-122a5403a336-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm\" (UID: \"b4ea9840-b602-4676-a8f2-122a5403a336\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:34:24.325644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.325624    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ea9840-b602-4676-a8f2-122a5403a336-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm\" (UID: \"b4ea9840-b602-4676-a8f2-122a5403a336\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:34:24.325691 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.325672    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4ea9840-b602-4676-a8f2-122a5403a336-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm\" (UID: \"b4ea9840-b602-4676-a8f2-122a5403a336\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:34:24.338994 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.338966    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:24.426496 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.426462    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ea9840-b602-4676-a8f2-122a5403a336-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm\" (UID: \"b4ea9840-b602-4676-a8f2-122a5403a336\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:34:24.426714 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.426502    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4ea9840-b602-4676-a8f2-122a5403a336-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm\" (UID: \"b4ea9840-b602-4676-a8f2-122a5403a336\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:34:24.426714 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.426527    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx257\" (UniqueName: \"kubernetes.io/projected/b4ea9840-b602-4676-a8f2-122a5403a336-kube-api-access-nx257\") pod \"maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm\" (UID: \"b4ea9840-b602-4676-a8f2-122a5403a336\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:34:24.426714 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.426546    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea9840-b602-4676-a8f2-122a5403a336-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm\" (UID: \"b4ea9840-b602-4676-a8f2-122a5403a336\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:34:24.426714 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:34:24.426651    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-serving-cert: secret "maas-api-e2e-shared-a-w3-5d8e85-serving-cert" not found
Sep 09 03:34:24.426909 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:34:24.426741    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ea9840-b602-4676-a8f2-122a5403a336-maas-api-tls podName:b4ea9840-b602-4676-a8f2-122a5403a336 nodeName:}" failed. No retries permitted until 2026-09-09 03:34:24.926720309 +0000 UTC m=+1282.063468152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b4ea9840-b602-4676-a8f2-122a5403a336-maas-api-tls") pod "maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm" (UID: "b4ea9840-b602-4676-a8f2-122a5403a336") : secret "maas-api-e2e-shared-a-w3-5d8e85-serving-cert" not found
Sep 09 03:34:24.427215 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.427194    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ea9840-b602-4676-a8f2-122a5403a336-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm\" (UID: \"b4ea9840-b602-4676-a8f2-122a5403a336\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:34:24.429108 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.429081    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea9840-b602-4676-a8f2-122a5403a336-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm\" (UID: \"b4ea9840-b602-4676-a8f2-122a5403a336\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:34:24.443167 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.443140    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx257\" (UniqueName: \"kubernetes.io/projected/b4ea9840-b602-4676-a8f2-122a5403a336-kube-api-access-nx257\") pod \"maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm\" (UID: \"b4ea9840-b602-4676-a8f2-122a5403a336\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:34:24.447105 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.447078    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:24.578025 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.577946    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:24.931371 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.931329    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4ea9840-b602-4676-a8f2-122a5403a336-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm\" (UID: \"b4ea9840-b602-4676-a8f2-122a5403a336\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:34:24.933538 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:24.933512    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4ea9840-b602-4676-a8f2-122a5403a336-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm\" (UID: \"b4ea9840-b602-4676-a8f2-122a5403a336\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:34:25.057709 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:25.057676    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:25.216738 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:25.216624    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:34:25.225258 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:25.225230    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:25.489333 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:25.489313    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"]
Sep 09 03:34:25.491617 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:34:25.491576    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ea9840_b602_4676_a8f2_122a5403a336.slice/crio-5c6dc55a09f5d036d96c60d80a54843d0bef49ffef78c9e1530485c7b3098571 WatchSource:0}: Error finding container 5c6dc55a09f5d036d96c60d80a54843d0bef49ffef78c9e1530485c7b3098571: Status 404 returned error can't find the container with id 5c6dc55a09f5d036d96c60d80a54843d0bef49ffef78c9e1530485c7b3098571
Sep 09 03:34:26.460258 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:26.460224    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm" event={"ID":"b4ea9840-b602-4676-a8f2-122a5403a336","Type":"ContainerStarted","Data":"b6910d441c4a465f02c12a3fbe9515b15c381fd2bfb48b78197d3c7d8dcbe43a"}
Sep 09 03:34:26.460527 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:26.460277    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm" event={"ID":"b4ea9840-b602-4676-a8f2-122a5403a336","Type":"ContainerStarted","Data":"5c6dc55a09f5d036d96c60d80a54843d0bef49ffef78c9e1530485c7b3098571"}
Sep 09 03:34:26.460527 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:26.460366    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:34:26.502321 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:26.502263    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm" podStartSLOduration=2.138501512 podStartE2EDuration="2.502243171s" totalImagesPullingTime="363.741659ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:34:26.501647575 +0000 UTC m=+1283.638395441" watchObservedRunningTime="2026-09-09 03:34:26.502243171 +0000 UTC m=+1283.638991037"
Sep 09 03:34:26.746995 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:26.746907    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:26.849622 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:26.849579    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:27.348289 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:27.348254    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:27.990040 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:27.990002    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:34:32.471217 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:32.471186    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:34:37.702044 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:37.702012    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"]
Sep 09 03:34:37.702527 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:37.702221    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz" podUID="a92a8360-f72c-4bfd-a200-36e13fa4c168" containerName="maas-api" containerID="cri-o://0cc7e64b398b3985a9444444cf9076d664d129ae1bb3f986581ceec4bfa5a5b5" gracePeriod=30
Sep 09 03:34:37.736884 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:37.736817    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:37.820769 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:37.820729    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:37.920023 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:37.919982    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:37.948650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:37.948624    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:34:38.056974 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.056891    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92a8360-f72c-4bfd-a200-36e13fa4c168-openshift-service-ca\") pod \"a92a8360-f72c-4bfd-a200-36e13fa4c168\" (UID: \"a92a8360-f72c-4bfd-a200-36e13fa4c168\") "
Sep 09 03:34:38.056974 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.056935    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fqxb\" (UniqueName: \"kubernetes.io/projected/a92a8360-f72c-4bfd-a200-36e13fa4c168-kube-api-access-9fqxb\") pod \"a92a8360-f72c-4bfd-a200-36e13fa4c168\" (UID: \"a92a8360-f72c-4bfd-a200-36e13fa4c168\") "
Sep 09 03:34:38.056974 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.056961    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:38.057260 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.057038    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92a8360-f72c-4bfd-a200-36e13fa4c168-metrics-certs\") pod \"a92a8360-f72c-4bfd-a200-36e13fa4c168\" (UID: \"a92a8360-f72c-4bfd-a200-36e13fa4c168\") "
Sep 09 03:34:38.057260 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.057077    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a92a8360-f72c-4bfd-a200-36e13fa4c168-maas-api-tls\") pod \"a92a8360-f72c-4bfd-a200-36e13fa4c168\" (UID: \"a92a8360-f72c-4bfd-a200-36e13fa4c168\") "
Sep 09 03:34:38.057378 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.057270    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a92a8360-f72c-4bfd-a200-36e13fa4c168-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a92a8360-f72c-4bfd-a200-36e13fa4c168" (UID: "a92a8360-f72c-4bfd-a200-36e13fa4c168"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:34:38.057466 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.057434    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92a8360-f72c-4bfd-a200-36e13fa4c168-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:34:38.059166 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.059140    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a92a8360-f72c-4bfd-a200-36e13fa4c168-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a92a8360-f72c-4bfd-a200-36e13fa4c168" (UID: "a92a8360-f72c-4bfd-a200-36e13fa4c168"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:34:38.059290 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.059170    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a92a8360-f72c-4bfd-a200-36e13fa4c168-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a92a8360-f72c-4bfd-a200-36e13fa4c168" (UID: "a92a8360-f72c-4bfd-a200-36e13fa4c168"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:34:38.059290 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.059182    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92a8360-f72c-4bfd-a200-36e13fa4c168-kube-api-access-9fqxb" (OuterVolumeSpecName: "kube-api-access-9fqxb") pod "a92a8360-f72c-4bfd-a200-36e13fa4c168" (UID: "a92a8360-f72c-4bfd-a200-36e13fa4c168"). InnerVolumeSpecName "kube-api-access-9fqxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:34:38.158478 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.158447    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92a8360-f72c-4bfd-a200-36e13fa4c168-metrics-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:34:38.158478 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.158471    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a92a8360-f72c-4bfd-a200-36e13fa4c168-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:34:38.158478 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.158480    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fqxb\" (UniqueName: \"kubernetes.io/projected/a92a8360-f72c-4bfd-a200-36e13fa4c168-kube-api-access-9fqxb\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:34:38.360501 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.360467    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:38.522066 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.522029    2560 generic.go:356] "Generic (PLEG): container finished" podID="a92a8360-f72c-4bfd-a200-36e13fa4c168" containerID="0cc7e64b398b3985a9444444cf9076d664d129ae1bb3f986581ceec4bfa5a5b5" exitCode=0
Sep 09 03:34:38.522247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.522073    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"
Sep 09 03:34:38.522247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.522077    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz" event={"ID":"a92a8360-f72c-4bfd-a200-36e13fa4c168","Type":"ContainerDied","Data":"0cc7e64b398b3985a9444444cf9076d664d129ae1bb3f986581ceec4bfa5a5b5"}
Sep 09 03:34:38.522247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.522127    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz" event={"ID":"a92a8360-f72c-4bfd-a200-36e13fa4c168","Type":"ContainerDied","Data":"39201b1331fb19764e32122c587753e5c1b89b097b17daf6a803ef3ecc2362d6"}
Sep 09 03:34:38.522247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.522156    2560 scope.go:117] "RemoveContainer" containerID="0cc7e64b398b3985a9444444cf9076d664d129ae1bb3f986581ceec4bfa5a5b5"
Sep 09 03:34:38.536853 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.536833    2560 scope.go:117] "RemoveContainer" containerID="0cc7e64b398b3985a9444444cf9076d664d129ae1bb3f986581ceec4bfa5a5b5"
Sep 09 03:34:38.537131 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:34:38.537110    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cc7e64b398b3985a9444444cf9076d664d129ae1bb3f986581ceec4bfa5a5b5\": container with ID starting with 0cc7e64b398b3985a9444444cf9076d664d129ae1bb3f986581ceec4bfa5a5b5 not found: ID does not exist" containerID="0cc7e64b398b3985a9444444cf9076d664d129ae1bb3f986581ceec4bfa5a5b5"
Sep 09 03:34:38.537169 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.537140    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc7e64b398b3985a9444444cf9076d664d129ae1bb3f986581ceec4bfa5a5b5"} err="failed to get container status \"0cc7e64b398b3985a9444444cf9076d664d129ae1bb3f986581ceec4bfa5a5b5\": rpc error: code = NotFound desc = could not find container \"0cc7e64b398b3985a9444444cf9076d664d129ae1bb3f986581ceec4bfa5a5b5\": container with ID starting with 0cc7e64b398b3985a9444444cf9076d664d129ae1bb3f986581ceec4bfa5a5b5 not found: ID does not exist"
Sep 09 03:34:38.564293 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.564261    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"]
Sep 09 03:34:38.574783 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:38.574758    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c44cc65a-f4ff8f47-r4fpz"]
Sep 09 03:34:39.359029 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:39.358995    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a92a8360-f72c-4bfd-a200-36e13fa4c168" path="/var/lib/kubelet/pods/a92a8360-f72c-4bfd-a200-36e13fa4c168/volumes"
Sep 09 03:34:43.597175 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:43.597141    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:34:44.562639 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:44.562581    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:44.634650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:44.634618    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:44.855074 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:44.855045    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:45.206354 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:45.206265    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:45.309129 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:45.309093    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:45.435524 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:45.435489    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:46.713946 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:46.713908    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:51.709019 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:51.708982    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:51.823869 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:51.823839    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:51.946851 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:51.946813    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:52.106836 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:52.106778    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:53.210787 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:53.210755    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:53.556985 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:53.556884    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:54.063079 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:54.063043    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:54.337187 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:54.337099    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:57.066458 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:57.066429    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:57.189551 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:57.189511    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:58.233044 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.233009    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"]
Sep 09 03:34:58.233426 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.233369    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a92a8360-f72c-4bfd-a200-36e13fa4c168" containerName="maas-api"
Sep 09 03:34:58.233426 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.233380    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a92a8360-f72c-4bfd-a200-36e13fa4c168" containerName="maas-api"
Sep 09 03:34:58.233504 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.233456    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a92a8360-f72c-4bfd-a200-36e13fa4c168" containerName="maas-api"
Sep 09 03:34:58.237961 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.237941    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:34:58.241391 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.241367    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-c34148-serving-cert\""
Sep 09 03:34:58.261321 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.261296    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"]
Sep 09 03:34:58.348640 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.348609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d710a0a8-d44b-41b0-865a-458e9fd88c4f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc\" (UID: \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:34:58.348812 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.348655    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d710a0a8-d44b-41b0-865a-458e9fd88c4f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc\" (UID: \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:34:58.348812 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.348774    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwfw4\" (UniqueName: \"kubernetes.io/projected/d710a0a8-d44b-41b0-865a-458e9fd88c4f-kube-api-access-zwfw4\") pod \"maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc\" (UID: \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:34:58.348812 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.348805    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d710a0a8-d44b-41b0-865a-458e9fd88c4f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc\" (UID: \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:34:58.369478 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.369451    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:58.450219 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.450186    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d710a0a8-d44b-41b0-865a-458e9fd88c4f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc\" (UID: \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:34:58.450401 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.450257    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfw4\" (UniqueName: \"kubernetes.io/projected/d710a0a8-d44b-41b0-865a-458e9fd88c4f-kube-api-access-zwfw4\") pod \"maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc\" (UID: \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:34:58.450401 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.450283    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d710a0a8-d44b-41b0-865a-458e9fd88c4f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc\" (UID: \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:34:58.450401 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.450342    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d710a0a8-d44b-41b0-865a-458e9fd88c4f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc\" (UID: \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:34:58.450625 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:34:58.450426    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-serving-cert: secret "maas-api-e2e-shared-b-w3-c34148-serving-cert" not found
Sep 09 03:34:58.450625 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:34:58.450498    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d710a0a8-d44b-41b0-865a-458e9fd88c4f-maas-api-tls podName:d710a0a8-d44b-41b0-865a-458e9fd88c4f nodeName:}" failed. No retries permitted until 2026-09-09 03:34:58.950480733 +0000 UTC m=+1316.087228595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d710a0a8-d44b-41b0-865a-458e9fd88c4f-maas-api-tls") pod "maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc" (UID: "d710a0a8-d44b-41b0-865a-458e9fd88c4f") : secret "maas-api-e2e-shared-b-w3-c34148-serving-cert" not found
Sep 09 03:34:58.451052 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.451034    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d710a0a8-d44b-41b0-865a-458e9fd88c4f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc\" (UID: \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:34:58.452699 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.452678    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d710a0a8-d44b-41b0-865a-458e9fd88c4f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc\" (UID: \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:34:58.463311 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.463284    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwfw4\" (UniqueName: \"kubernetes.io/projected/d710a0a8-d44b-41b0-865a-458e9fd88c4f-kube-api-access-zwfw4\") pod \"maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc\" (UID: \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:34:58.505463 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.505383    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:58.629710 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.629676    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:58.956659 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.956621    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d710a0a8-d44b-41b0-865a-458e9fd88c4f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc\" (UID: \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:34:58.959137 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:58.959101    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d710a0a8-d44b-41b0-865a-458e9fd88c4f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc\" (UID: \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:34:59.148026 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:59.147988    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:34:59.291357 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:59.291280    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:59.468404 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:59.468375    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"]
Sep 09 03:34:59.469174 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:34:59.469149    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd710a0a8_d44b_41b0_865a_458e9fd88c4f.slice/crio-749cc07dea21aae8a9221796cc788955183dd107f8b8352961e5be33f263e2ad WatchSource:0}: Error finding container 749cc07dea21aae8a9221796cc788955183dd107f8b8352961e5be33f263e2ad: Status 404 returned error can't find the container with id 749cc07dea21aae8a9221796cc788955183dd107f8b8352961e5be33f263e2ad
Sep 09 03:34:59.524705 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:59.524674    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:34:59.629210 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:34:59.629178    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc" event={"ID":"d710a0a8-d44b-41b0-865a-458e9fd88c4f","Type":"ContainerStarted","Data":"749cc07dea21aae8a9221796cc788955183dd107f8b8352961e5be33f263e2ad"}
Sep 09 03:35:00.636356 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:00.636323    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc" event={"ID":"d710a0a8-d44b-41b0-865a-458e9fd88c4f","Type":"ContainerStarted","Data":"6cbdcc3822e54c41c0526d2945018d903986a7e0318c6ba6ac5983e9d8d367c1"}
Sep 09 03:35:00.636738 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:00.636432    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:35:00.665778 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:00.665730    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc" podStartSLOduration=2.074155082 podStartE2EDuration="2.665712771s" totalImagesPullingTime="591.557689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:34:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:35:00.663191901 +0000 UTC m=+1317.799939776" watchObservedRunningTime="2026-09-09 03:35:00.665712771 +0000 UTC m=+1317.802460629"
Sep 09 03:35:01.624516 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:01.624479    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:01.725004 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:01.724967    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:02.631109 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:02.631073    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:03.012671 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:03.012555    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:03.093924 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:03.093882    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:35:03.157721 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:03.157673    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:06.338989 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:06.338955    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:06.456050 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:06.456018    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:06.646912 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:06.646879    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:35:07.509844 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:07.509804    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"]
Sep 09 03:35:07.515366 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:07.515339    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:35:07.518889 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:07.518863    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-eca017-serving-cert\""
Sep 09 03:35:07.527518 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:07.527485    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"]
Sep 09 03:35:07.534996 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:07.534971    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsghl\" (UniqueName: \"kubernetes.io/projected/963cd7f1-7fed-459a-8dff-96f5b992e31b-kube-api-access-zsghl\") pod \"maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb\" (UID: \"963cd7f1-7fed-459a-8dff-96f5b992e31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:35:07.535154 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:07.535019    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963cd7f1-7fed-459a-8dff-96f5b992e31b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb\" (UID: \"963cd7f1-7fed-459a-8dff-96f5b992e31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:35:07.535154 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:07.535131    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963cd7f1-7fed-459a-8dff-96f5b992e31b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb\" (UID: \"963cd7f1-7fed-459a-8dff-96f5b992e31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:35:07.535267 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:07.535184    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963cd7f1-7fed-459a-8dff-96f5b992e31b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb\" (UID: \"963cd7f1-7fed-459a-8dff-96f5b992e31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:35:07.555829 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:07.555774    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:07.636200 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:07.636156    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963cd7f1-7fed-459a-8dff-96f5b992e31b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb\" (UID: \"963cd7f1-7fed-459a-8dff-96f5b992e31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:35:07.636403 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:07.636232    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963cd7f1-7fed-459a-8dff-96f5b992e31b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb\" (UID: \"963cd7f1-7fed-459a-8dff-96f5b992e31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:35:07.636403 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:07.636281    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963cd7f1-7fed-459a-8dff-96f5b992e31b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb\" (UID: \"963cd7f1-7fed-459a-8dff-96f5b992e31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:35:07.636403 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:07.636345    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsghl\" (UniqueName: \"kubernetes.io/projected/963cd7f1-7fed-459a-8dff-96f5b992e31b-kube-api-access-zsghl\") pod \"maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb\" (UID: \"963cd7f1-7fed-459a-8dff-96f5b992e31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:35:07.636576 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:35:07.636443    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-serving-cert: secret "maas-api-e2e-shared-a-w2-eca017-serving-cert" not found
Sep 09 03:35:07.636576 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:35:07.636527    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/963cd7f1-7fed-459a-8dff-96f5b992e31b-maas-api-tls podName:963cd7f1-7fed-459a-8dff-96f5b992e31b nodeName:}" failed. No retries permitted until 2026-09-09 03:35:08.136509057 +0000 UTC m=+1325.273256903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/963cd7f1-7fed-459a-8dff-96f5b992e31b-maas-api-tls") pod "maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb" (UID: "963cd7f1-7fed-459a-8dff-96f5b992e31b") : secret "maas-api-e2e-shared-a-w2-eca017-serving-cert" not found
Sep 09 03:35:07.637006 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:07.636984    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963cd7f1-7fed-459a-8dff-96f5b992e31b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb\" (UID: \"963cd7f1-7fed-459a-8dff-96f5b992e31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:35:07.638718 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:07.638701    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963cd7f1-7fed-459a-8dff-96f5b992e31b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb\" (UID: \"963cd7f1-7fed-459a-8dff-96f5b992e31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:35:07.659891 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:07.659854    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsghl\" (UniqueName: \"kubernetes.io/projected/963cd7f1-7fed-459a-8dff-96f5b992e31b-kube-api-access-zsghl\") pod \"maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb\" (UID: \"963cd7f1-7fed-459a-8dff-96f5b992e31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:35:07.677040 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:07.677002    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:07.762659 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:07.762571    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:08.108063 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:08.108031    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:08.142506 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:08.142462    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963cd7f1-7fed-459a-8dff-96f5b992e31b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb\" (UID: \"963cd7f1-7fed-459a-8dff-96f5b992e31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:35:08.145020 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:08.144990    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963cd7f1-7fed-459a-8dff-96f5b992e31b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb\" (UID: \"963cd7f1-7fed-459a-8dff-96f5b992e31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:35:08.203969 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:08.203925    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:08.427840 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:08.427738    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:35:08.705305 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:08.705278    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"]
Sep 09 03:35:08.707182 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:35:08.707152    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963cd7f1_7fed_459a_8dff_96f5b992e31b.slice/crio-9039aeed27604f8b130671c54e28a640d4ab017655f4e10d691b97d832f0f70c WatchSource:0}: Error finding container 9039aeed27604f8b130671c54e28a640d4ab017655f4e10d691b97d832f0f70c: Status 404 returned error can't find the container with id 9039aeed27604f8b130671c54e28a640d4ab017655f4e10d691b97d832f0f70c
Sep 09 03:35:09.684997 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:09.684957    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb" event={"ID":"963cd7f1-7fed-459a-8dff-96f5b992e31b","Type":"ContainerStarted","Data":"c27c3d5bd54af095d3efa1193a09009664855c56e7412576bc74aaeff43fe0e8"}
Sep 09 03:35:09.685185 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:09.685014    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb" event={"ID":"963cd7f1-7fed-459a-8dff-96f5b992e31b","Type":"ContainerStarted","Data":"9039aeed27604f8b130671c54e28a640d4ab017655f4e10d691b97d832f0f70c"}
Sep 09 03:35:09.685185 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:09.685058    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:35:09.707625 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:09.707550    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb" podStartSLOduration=2.214532009 podStartE2EDuration="2.70753276s" totalImagesPullingTime="493.000751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:35:09.705627243 +0000 UTC m=+1326.842375109" watchObservedRunningTime="2026-09-09 03:35:09.70753276 +0000 UTC m=+1326.844280641"
Sep 09 03:35:12.799224 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:12.799190    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:35:15.696855 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:15.696824    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:35:20.265430 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:20.265396    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:21.999138 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:21.999087    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:25.144450 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:25.144417    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:30.005889 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:30.005795    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:30.301569 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:30.301481    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:35:31.384015 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:31.383986    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:31.722738 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:31.722650    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:32.074679 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:32.074571    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:32.252682 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:32.252641    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:35.729922 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:35.729882    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:35.846955 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:35.846922    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:36.999190 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:36.999161    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"]
Sep 09 03:35:37.002247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.002228    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:35:37.005079 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.005061    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-0d0cae-serving-cert\""
Sep 09 03:35:37.019128 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.019105    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"]
Sep 09 03:35:37.025950 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.025916    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5413b949-e629-4b13-878c-6ea8480c1cf8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx\" (UID: \"5413b949-e629-4b13-878c-6ea8480c1cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:35:37.026086 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.025958    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5413b949-e629-4b13-878c-6ea8480c1cf8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx\" (UID: \"5413b949-e629-4b13-878c-6ea8480c1cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:35:37.026086 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.025989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5413b949-e629-4b13-878c-6ea8480c1cf8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx\" (UID: \"5413b949-e629-4b13-878c-6ea8480c1cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:35:37.026086 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.026079    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4djf\" (UniqueName: \"kubernetes.io/projected/5413b949-e629-4b13-878c-6ea8480c1cf8-kube-api-access-q4djf\") pod \"maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx\" (UID: \"5413b949-e629-4b13-878c-6ea8480c1cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:35:37.031707 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.031683    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:37.126680 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.126651    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4djf\" (UniqueName: \"kubernetes.io/projected/5413b949-e629-4b13-878c-6ea8480c1cf8-kube-api-access-q4djf\") pod \"maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx\" (UID: \"5413b949-e629-4b13-878c-6ea8480c1cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:35:37.126853 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.126706    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5413b949-e629-4b13-878c-6ea8480c1cf8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx\" (UID: \"5413b949-e629-4b13-878c-6ea8480c1cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:35:37.126853 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.126741    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5413b949-e629-4b13-878c-6ea8480c1cf8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx\" (UID: \"5413b949-e629-4b13-878c-6ea8480c1cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:35:37.126853 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.126777    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5413b949-e629-4b13-878c-6ea8480c1cf8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx\" (UID: \"5413b949-e629-4b13-878c-6ea8480c1cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:35:37.126853 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:35:37.126821    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-serving-cert: secret "maas-api-e2e-shared-b-w2-0d0cae-serving-cert" not found
Sep 09 03:35:37.127071 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:35:37.126881    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5413b949-e629-4b13-878c-6ea8480c1cf8-maas-api-tls podName:5413b949-e629-4b13-878c-6ea8480c1cf8 nodeName:}" failed. No retries permitted until 2026-09-09 03:35:37.626860134 +0000 UTC m=+1354.763607986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5413b949-e629-4b13-878c-6ea8480c1cf8-maas-api-tls") pod "maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx" (UID: "5413b949-e629-4b13-878c-6ea8480c1cf8") : secret "maas-api-e2e-shared-b-w2-0d0cae-serving-cert" not found
Sep 09 03:35:37.127483 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.127457    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5413b949-e629-4b13-878c-6ea8480c1cf8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx\" (UID: \"5413b949-e629-4b13-878c-6ea8480c1cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:35:37.129173 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.129154    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5413b949-e629-4b13-878c-6ea8480c1cf8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx\" (UID: \"5413b949-e629-4b13-878c-6ea8480c1cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:35:37.135554 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.135533    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:37.139205 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.139188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4djf\" (UniqueName: \"kubernetes.io/projected/5413b949-e629-4b13-878c-6ea8480c1cf8-kube-api-access-q4djf\") pod \"maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx\" (UID: \"5413b949-e629-4b13-878c-6ea8480c1cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:35:37.314576 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.314484    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:37.610607 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.610568    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:37.631949 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.631923    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5413b949-e629-4b13-878c-6ea8480c1cf8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx\" (UID: \"5413b949-e629-4b13-878c-6ea8480c1cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:35:37.634393 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.634372    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5413b949-e629-4b13-878c-6ea8480c1cf8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx\" (UID: \"5413b949-e629-4b13-878c-6ea8480c1cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:35:37.709406 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.709373    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:37.870046 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.869970    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb"]
Sep 09 03:35:37.873773 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.873753    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb"
Sep 09 03:35:37.878313 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.878293    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-0d0cae-dockercfg-mxv74\""
Sep 09 03:35:37.878925 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.878905    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-0d0cae\""
Sep 09 03:35:37.899485 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.899459    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb"]
Sep 09 03:35:37.912188 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.912168    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:35:37.935120 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.935089    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmspx\" (UniqueName: \"kubernetes.io/projected/2f1e0d6e-d9bb-4125-a8a6-f26da5242731-kube-api-access-hmspx\") pod \"payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb\" (UID: \"2f1e0d6e-d9bb-4125-a8a6-f26da5242731\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb"
Sep 09 03:35:37.935234 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:37.935209    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1e0d6e-d9bb-4125-a8a6-f26da5242731-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb\" (UID: \"2f1e0d6e-d9bb-4125-a8a6-f26da5242731\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb"
Sep 09 03:35:38.035847 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:38.035809    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmspx\" (UniqueName: \"kubernetes.io/projected/2f1e0d6e-d9bb-4125-a8a6-f26da5242731-kube-api-access-hmspx\") pod \"payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb\" (UID: \"2f1e0d6e-d9bb-4125-a8a6-f26da5242731\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb"
Sep 09 03:35:38.036278 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:38.035874    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1e0d6e-d9bb-4125-a8a6-f26da5242731-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb\" (UID: \"2f1e0d6e-d9bb-4125-a8a6-f26da5242731\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb"
Sep 09 03:35:38.036692 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:38.036669    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1e0d6e-d9bb-4125-a8a6-f26da5242731-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb\" (UID: \"2f1e0d6e-d9bb-4125-a8a6-f26da5242731\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb"
Sep 09 03:35:38.045525 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:38.045504    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmspx\" (UniqueName: \"kubernetes.io/projected/2f1e0d6e-d9bb-4125-a8a6-f26da5242731-kube-api-access-hmspx\") pod \"payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb\" (UID: \"2f1e0d6e-d9bb-4125-a8a6-f26da5242731\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb"
Sep 09 03:35:38.184755 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:38.184679    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb"
Sep 09 03:35:38.191188 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:38.191166    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"]
Sep 09 03:35:38.195336 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:35:38.195307    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5413b949_e629_4b13_878c_6ea8480c1cf8.slice/crio-9233ab9e7df33dcbd121b4fc65bcfc3cf6d4d3d86aab965625e19c5e726e01d6 WatchSource:0}: Error finding container 9233ab9e7df33dcbd121b4fc65bcfc3cf6d4d3d86aab965625e19c5e726e01d6: Status 404 returned error can't find the container with id 9233ab9e7df33dcbd121b4fc65bcfc3cf6d4d3d86aab965625e19c5e726e01d6
Sep 09 03:35:38.463107 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:38.463050    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb"]
Sep 09 03:35:38.465397 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:35:38.465369    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1e0d6e_d9bb_4125_a8a6_f26da5242731.slice/crio-3fed1a00814a72d3c7152367651941ce7d1854c753a0c86eb993795d405a05a0 WatchSource:0}: Error finding container 3fed1a00814a72d3c7152367651941ce7d1854c753a0c86eb993795d405a05a0: Status 404 returned error can't find the container with id 3fed1a00814a72d3c7152367651941ce7d1854c753a0c86eb993795d405a05a0
Sep 09 03:35:38.843769 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:38.843678    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb" event={"ID":"2f1e0d6e-d9bb-4125-a8a6-f26da5242731","Type":"ContainerStarted","Data":"3c211a05d7db90f2fd96b208fdfceaeb8c8bf170aa7407e5c8d2b46a1ac4cadb"}
Sep 09 03:35:38.843769 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:38.843756    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb" event={"ID":"2f1e0d6e-d9bb-4125-a8a6-f26da5242731","Type":"ContainerStarted","Data":"3fed1a00814a72d3c7152367651941ce7d1854c753a0c86eb993795d405a05a0"}
Sep 09 03:35:38.844006 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:38.843959    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb"
Sep 09 03:35:38.845841 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:38.845814    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx" event={"ID":"5413b949-e629-4b13-878c-6ea8480c1cf8","Type":"ContainerStarted","Data":"306aa7bd73340688956ad250afafeabd494f99341b2070770b5f93c4e983ada6"}
Sep 09 03:35:38.845962 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:38.845862    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx" event={"ID":"5413b949-e629-4b13-878c-6ea8480c1cf8","Type":"ContainerStarted","Data":"9233ab9e7df33dcbd121b4fc65bcfc3cf6d4d3d86aab965625e19c5e726e01d6"}
Sep 09 03:35:38.845962 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:38.845893    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:35:38.862518 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:38.862468    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb" podStartSLOduration=1.8624491189999999 podStartE2EDuration="1.862449119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:35:38.861097133 +0000 UTC m=+1355.997845002" watchObservedRunningTime="2026-09-09 03:35:38.862449119 +0000 UTC m=+1355.999196989"
Sep 09 03:35:38.884039 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:38.883989    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx" podStartSLOduration=2.464086291 podStartE2EDuration="2.883974192s" totalImagesPullingTime="419.887901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:35:38.88303691 +0000 UTC m=+1356.019784777" watchObservedRunningTime="2026-09-09 03:35:38.883974192 +0000 UTC m=+1356.020722057"
Sep 09 03:35:39.279907 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:39.279878    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:43.996899 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:43.996862    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:35:44.856568 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:44.856536    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:35:49.853138 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:49.853106    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb"
Sep 09 03:35:51.550540 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:51.550500    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:35:59.212792 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:35:59.212759    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:36:00.598395 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:00.598359    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:03.529339 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:03.529302    2560 scope.go:117] "RemoveContainer" containerID="764e39c0e77adee4d9d0967f8f1ceadc2b4466010adab323e6668d0c39e05f93"
Sep 09 03:36:03.537668 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:03.537643    2560 scope.go:117] "RemoveContainer" containerID="9dc8e7de7c498a853946a882b3fa5c9129d3120b2ab2858923613c19e4712749"
Sep 09 03:36:10.148807 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:10.148766    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7647-139627984443408"]
Sep 09 03:36:10.153960 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:10.153941    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-139627984443408"
Sep 09 03:36:10.157172 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:10.157143    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-k4rsk\""
Sep 09 03:36:10.159569 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:10.159546    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7647-139627984443408"]
Sep 09 03:36:10.342820 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:10.342781    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b495w\" (UniqueName: \"kubernetes.io/projected/bdd812d4-ab1e-495c-97dc-4ec0617dd22b-kube-api-access-b495w\") pod \"test-curl-7647-139627984443408\" (UID: \"bdd812d4-ab1e-495c-97dc-4ec0617dd22b\") " pod="opendatahub/test-curl-7647-139627984443408"
Sep 09 03:36:10.444263 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:10.444185    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b495w\" (UniqueName: \"kubernetes.io/projected/bdd812d4-ab1e-495c-97dc-4ec0617dd22b-kube-api-access-b495w\") pod \"test-curl-7647-139627984443408\" (UID: \"bdd812d4-ab1e-495c-97dc-4ec0617dd22b\") " pod="opendatahub/test-curl-7647-139627984443408"
Sep 09 03:36:10.453901 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:10.453869    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b495w\" (UniqueName: \"kubernetes.io/projected/bdd812d4-ab1e-495c-97dc-4ec0617dd22b-kube-api-access-b495w\") pod \"test-curl-7647-139627984443408\" (UID: \"bdd812d4-ab1e-495c-97dc-4ec0617dd22b\") " pod="opendatahub/test-curl-7647-139627984443408"
Sep 09 03:36:10.464697 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:10.464676    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-139627984443408"
Sep 09 03:36:10.734047 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:10.734024    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7647-139627984443408"]
Sep 09 03:36:10.735925 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:36:10.735889    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdd812d4_ab1e_495c_97dc_4ec0617dd22b.slice/crio-80ed61b8bd55bdebc6b54128f36f68b03b9ea9d8c15f4e40a2933815818db42d WatchSource:0}: Error finding container 80ed61b8bd55bdebc6b54128f36f68b03b9ea9d8c15f4e40a2933815818db42d: Status 404 returned error can't find the container with id 80ed61b8bd55bdebc6b54128f36f68b03b9ea9d8c15f4e40a2933815818db42d
Sep 09 03:36:11.015474 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:11.015382    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7647-139627984443408" event={"ID":"bdd812d4-ab1e-495c-97dc-4ec0617dd22b","Type":"ContainerStarted","Data":"80ed61b8bd55bdebc6b54128f36f68b03b9ea9d8c15f4e40a2933815818db42d"}
Sep 09 03:36:12.022043 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:12.022009    2560 generic.go:356] "Generic (PLEG): container finished" podID="bdd812d4-ab1e-495c-97dc-4ec0617dd22b" containerID="139e67c7cdb444f62b2ba2ee170632b63a8ae83467617b1e3bde482c083952e4" exitCode=0
Sep 09 03:36:12.022447 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:12.022101    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7647-139627984443408" event={"ID":"bdd812d4-ab1e-495c-97dc-4ec0617dd22b","Type":"ContainerDied","Data":"139e67c7cdb444f62b2ba2ee170632b63a8ae83467617b1e3bde482c083952e4"}
Sep 09 03:36:13.156803 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:13.156777    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-139627984443408"
Sep 09 03:36:13.189676 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:13.189651    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7647-139627984443408_bdd812d4-ab1e-495c-97dc-4ec0617dd22b/test-curl-7647-139627984443408/0.log"
Sep 09 03:36:13.218628 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:13.218581    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7647-139627984443408"]
Sep 09 03:36:13.229290 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:13.229269    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7647-139627984443408"]
Sep 09 03:36:13.270954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:13.270932    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b495w\" (UniqueName: \"kubernetes.io/projected/bdd812d4-ab1e-495c-97dc-4ec0617dd22b-kube-api-access-b495w\") pod \"bdd812d4-ab1e-495c-97dc-4ec0617dd22b\" (UID: \"bdd812d4-ab1e-495c-97dc-4ec0617dd22b\") "
Sep 09 03:36:13.272909 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:13.272887    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd812d4-ab1e-495c-97dc-4ec0617dd22b-kube-api-access-b495w" (OuterVolumeSpecName: "kube-api-access-b495w") pod "bdd812d4-ab1e-495c-97dc-4ec0617dd22b" (UID: "bdd812d4-ab1e-495c-97dc-4ec0617dd22b"). InnerVolumeSpecName "kube-api-access-b495w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:36:13.359712 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:13.359683    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdd812d4-ab1e-495c-97dc-4ec0617dd22b" path="/var/lib/kubelet/pods/bdd812d4-ab1e-495c-97dc-4ec0617dd22b/volumes"
Sep 09 03:36:13.371833 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:13.371811    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b495w\" (UniqueName: \"kubernetes.io/projected/bdd812d4-ab1e-495c-97dc-4ec0617dd22b-kube-api-access-b495w\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:36:13.860302 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:13.860250    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:14.035215 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:14.035185    2560 scope.go:117] "RemoveContainer" containerID="139e67c7cdb444f62b2ba2ee170632b63a8ae83467617b1e3bde482c083952e4"
Sep 09 03:36:14.035215 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:14.035208    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-139627984443408"
Sep 09 03:36:14.201708 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:14.201629    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:14.380541 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:14.380510    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:16.895233 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:16.895200    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:16.991720 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:16.991685    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:17.823923 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:17.823895    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:18.627442 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.627405    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"]
Sep 09 03:36:18.627925 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.627819    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd812d4-ab1e-495c-97dc-4ec0617dd22b" containerName="test-curl-7647-139627984443408"
Sep 09 03:36:18.627925 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.627832    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd812d4-ab1e-495c-97dc-4ec0617dd22b" containerName="test-curl-7647-139627984443408"
Sep 09 03:36:18.627925 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.627905    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdd812d4-ab1e-495c-97dc-4ec0617dd22b" containerName="test-curl-7647-139627984443408"
Sep 09 03:36:18.632329 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.632309    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:36:18.637145 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.637123    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-123302cc-serving-cert\""
Sep 09 03:36:18.645348 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.645325    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"]
Sep 09 03:36:18.664757 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.664731    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:18.726657 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.726616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm7tn\" (UniqueName: \"kubernetes.io/projected/410f304a-1734-433d-9a40-ab92dd7c303f-kube-api-access-lm7tn\") pod \"maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp\" (UID: \"410f304a-1734-433d-9a40-ab92dd7c303f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:36:18.726657 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.726659    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410f304a-1734-433d-9a40-ab92dd7c303f-metrics-certs\") pod \"maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp\" (UID: \"410f304a-1734-433d-9a40-ab92dd7c303f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:36:18.726898 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.726708    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/410f304a-1734-433d-9a40-ab92dd7c303f-openshift-service-ca\") pod \"maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp\" (UID: \"410f304a-1734-433d-9a40-ab92dd7c303f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:36:18.726898 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.726774    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/410f304a-1734-433d-9a40-ab92dd7c303f-maas-api-tls\") pod \"maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp\" (UID: \"410f304a-1734-433d-9a40-ab92dd7c303f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:36:18.761235 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.761198    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:18.827535 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.827492    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/410f304a-1734-433d-9a40-ab92dd7c303f-openshift-service-ca\") pod \"maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp\" (UID: \"410f304a-1734-433d-9a40-ab92dd7c303f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:36:18.827535 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.827535    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/410f304a-1734-433d-9a40-ab92dd7c303f-maas-api-tls\") pod \"maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp\" (UID: \"410f304a-1734-433d-9a40-ab92dd7c303f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:36:18.827814 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.827676    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm7tn\" (UniqueName: \"kubernetes.io/projected/410f304a-1734-433d-9a40-ab92dd7c303f-kube-api-access-lm7tn\") pod \"maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp\" (UID: \"410f304a-1734-433d-9a40-ab92dd7c303f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:36:18.827814 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.827707    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410f304a-1734-433d-9a40-ab92dd7c303f-metrics-certs\") pod \"maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp\" (UID: \"410f304a-1734-433d-9a40-ab92dd7c303f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:36:18.827814 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:36:18.827729    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-serving-cert: secret "maas-api-e2e-mt-123302cc-serving-cert" not found
Sep 09 03:36:18.827814 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:36:18.827797    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/410f304a-1734-433d-9a40-ab92dd7c303f-maas-api-tls podName:410f304a-1734-433d-9a40-ab92dd7c303f nodeName:}" failed. No retries permitted until 2026-09-09 03:36:19.32777709 +0000 UTC m=+1396.464524947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/410f304a-1734-433d-9a40-ab92dd7c303f-maas-api-tls") pod "maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp" (UID: "410f304a-1734-433d-9a40-ab92dd7c303f") : secret "maas-api-e2e-mt-123302cc-serving-cert" not found
Sep 09 03:36:18.828354 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.828306    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/410f304a-1734-433d-9a40-ab92dd7c303f-openshift-service-ca\") pod \"maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp\" (UID: \"410f304a-1734-433d-9a40-ab92dd7c303f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:36:18.830370 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.830346    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410f304a-1734-433d-9a40-ab92dd7c303f-metrics-certs\") pod \"maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp\" (UID: \"410f304a-1734-433d-9a40-ab92dd7c303f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:36:18.841895 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:18.841874    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm7tn\" (UniqueName: \"kubernetes.io/projected/410f304a-1734-433d-9a40-ab92dd7c303f-kube-api-access-lm7tn\") pod \"maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp\" (UID: \"410f304a-1734-433d-9a40-ab92dd7c303f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:36:19.010942 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:19.010841    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:36:19.011110 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:19.011041    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:19.237535 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:19.237500    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:19.332978 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:19.332890    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/410f304a-1734-433d-9a40-ab92dd7c303f-maas-api-tls\") pod \"maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp\" (UID: \"410f304a-1734-433d-9a40-ab92dd7c303f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:36:19.335416 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:19.335389    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/410f304a-1734-433d-9a40-ab92dd7c303f-maas-api-tls\") pod \"maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp\" (UID: \"410f304a-1734-433d-9a40-ab92dd7c303f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:36:19.342305 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:19.342275    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:19.545517 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:19.545478    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:36:19.818011 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:19.817984    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"]
Sep 09 03:36:19.819580 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:36:19.819552    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod410f304a_1734_433d_9a40_ab92dd7c303f.slice/crio-23b3d1aad40b3994b34db0109286c3fb804068bb4c0a1e0504b1854098f2458a WatchSource:0}: Error finding container 23b3d1aad40b3994b34db0109286c3fb804068bb4c0a1e0504b1854098f2458a: Status 404 returned error can't find the container with id 23b3d1aad40b3994b34db0109286c3fb804068bb4c0a1e0504b1854098f2458a
Sep 09 03:36:20.068726 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:20.068685    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp" event={"ID":"410f304a-1734-433d-9a40-ab92dd7c303f","Type":"ContainerStarted","Data":"23b3d1aad40b3994b34db0109286c3fb804068bb4c0a1e0504b1854098f2458a"}
Sep 09 03:36:21.075694 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:21.075652    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp" event={"ID":"410f304a-1734-433d-9a40-ab92dd7c303f","Type":"ContainerStarted","Data":"b1085efb676150742c158d39a8efca67c04d4c70079deb61edc6497a04a50ec2"}
Sep 09 03:36:21.076091 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:21.075737    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:36:21.096028 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:21.095980    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp" podStartSLOduration=2.614975036 podStartE2EDuration="3.095966451s" totalImagesPullingTime="480.991415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:36:21.093372997 +0000 UTC m=+1398.230120880" watchObservedRunningTime="2026-09-09 03:36:21.095966451 +0000 UTC m=+1398.232714317"
Sep 09 03:36:27.086717 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:27.086681    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:36:40.704913 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:40.704883    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:36:41.900991 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:41.900954    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:48.291737 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.291702    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"]
Sep 09 03:36:48.295303 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.295281    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.300184 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.300158    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-123302cc\"/\"kube-root-ca.crt\""
Sep 09 03:36:48.300305 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.300166    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-123302cc\"/\"default-dockercfg-87tc5\""
Sep 09 03:36:48.300305 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.300258    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-123302cc\"/\"openshift-service-ca.crt\""
Sep 09 03:36:48.300305 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.300267    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-123302cc\"/\"e2e-lifecycle-model-123302cc-kserve-self-signed-certs\""
Sep 09 03:36:48.306150 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.306131    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"]
Sep 09 03:36:48.312374 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.312117    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-dshm\") pod \"e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") " pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.312374 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.312209    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-home\") pod \"e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") " pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.312374 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.312268    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-model-cache\") pod \"e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") " pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.312374 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.312315    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-tmp-dir\") pod \"e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") " pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.312775 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.312377    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x72gp\" (UniqueName: \"kubernetes.io/projected/3f442ec8-955c-4e4e-abd3-de1f6906f528-kube-api-access-x72gp\") pod \"e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") " pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.312775 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.312506    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f442ec8-955c-4e4e-abd3-de1f6906f528-tls-certs\") pod \"e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") " pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.315915 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.315895    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:36:48.412910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.412884    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-home\") pod \"e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") " pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.413055 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.412924    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-model-cache\") pod \"e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") " pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.413055 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.412950    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-tmp-dir\") pod \"e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") " pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.413161 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.413062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x72gp\" (UniqueName: \"kubernetes.io/projected/3f442ec8-955c-4e4e-abd3-de1f6906f528-kube-api-access-x72gp\") pod \"e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") " pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.413161 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.413111    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f442ec8-955c-4e4e-abd3-de1f6906f528-tls-certs\") pod \"e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") " pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.413280 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.413159    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-dshm\") pod \"e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") " pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.413280 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.413263    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-tmp-dir\") pod \"e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") " pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.413388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.413301    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-home\") pod \"e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") " pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.413388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.413361    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-model-cache\") pod \"e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") " pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.415372 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.415352    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-dshm\") pod \"e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") " pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.415529 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.415510    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f442ec8-955c-4e4e-abd3-de1f6906f528-tls-certs\") pod \"e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") " pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.423033 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.423012    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x72gp\" (UniqueName: \"kubernetes.io/projected/3f442ec8-955c-4e4e-abd3-de1f6906f528-kube-api-access-x72gp\") pod \"e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") " pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.605235 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.605197    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:48.874691 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:48.874663    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"]
Sep 09 03:36:48.876568 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:36:48.876542    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f442ec8_955c_4e4e_abd3_de1f6906f528.slice/crio-07bffc880878987497df37985eccd20d7d8d71efec518f79eafdcfbef9368f0d WatchSource:0}: Error finding container 07bffc880878987497df37985eccd20d7d8d71efec518f79eafdcfbef9368f0d: Status 404 returned error can't find the container with id 07bffc880878987497df37985eccd20d7d8d71efec518f79eafdcfbef9368f0d
Sep 09 03:36:49.227199 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:49.227096    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc" event={"ID":"3f442ec8-955c-4e4e-abd3-de1f6906f528","Type":"ContainerStarted","Data":"bf8d243765f6e57e7f5366747030f7564206d63f6feda308dd3626a7cfef2bce"}
Sep 09 03:36:49.227199 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:49.227166    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc" event={"ID":"3f442ec8-955c-4e4e-abd3-de1f6906f528","Type":"ContainerStarted","Data":"07bffc880878987497df37985eccd20d7d8d71efec518f79eafdcfbef9368f0d"}
Sep 09 03:36:49.249661 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:49.249609    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc" podStartSLOduration=1.249583189 podStartE2EDuration="1.249583189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:36:49.246197967 +0000 UTC m=+1426.382945854" watchObservedRunningTime="2026-09-09 03:36:49.249583189 +0000 UTC m=+1426.386331055"
Sep 09 03:36:56.099988 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:56.099910    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:36:58.605903 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:58.605865    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:58.605903 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:58.605911    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:58.618469 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:58.618442    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:36:59.293004 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:36:59.292972    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:37:03.560245 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:03.560215    2560 scope.go:117] "RemoveContainer" containerID="f09d76b7712f3b846de938f0340c283c6a6e74c02dccca06c03f028e3b999a79"
Sep 09 03:37:15.296392 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:15.296351    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:37:32.509122 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:32.509092    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:37:37.173832 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.173798    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"]
Sep 09 03:37:37.174210 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.174070    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp" podUID="410f304a-1734-433d-9a40-ab92dd7c303f" containerName="maas-api" containerID="cri-o://b1085efb676150742c158d39a8efca67c04d4c70079deb61edc6497a04a50ec2" gracePeriod=30
Sep 09 03:37:37.421936 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.421901    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:37:37.478741 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.478646    2560 generic.go:356] "Generic (PLEG): container finished" podID="410f304a-1734-433d-9a40-ab92dd7c303f" containerID="b1085efb676150742c158d39a8efca67c04d4c70079deb61edc6497a04a50ec2" exitCode=0
Sep 09 03:37:37.478741 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.478689    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"
Sep 09 03:37:37.478741 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.478728    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp" event={"ID":"410f304a-1734-433d-9a40-ab92dd7c303f","Type":"ContainerDied","Data":"b1085efb676150742c158d39a8efca67c04d4c70079deb61edc6497a04a50ec2"}
Sep 09 03:37:37.479011 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.478786    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp" event={"ID":"410f304a-1734-433d-9a40-ab92dd7c303f","Type":"ContainerDied","Data":"23b3d1aad40b3994b34db0109286c3fb804068bb4c0a1e0504b1854098f2458a"}
Sep 09 03:37:37.479011 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.478815    2560 scope.go:117] "RemoveContainer" containerID="b1085efb676150742c158d39a8efca67c04d4c70079deb61edc6497a04a50ec2"
Sep 09 03:37:37.487855 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.487840    2560 scope.go:117] "RemoveContainer" containerID="b1085efb676150742c158d39a8efca67c04d4c70079deb61edc6497a04a50ec2"
Sep 09 03:37:37.488103 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:37:37.488085    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1085efb676150742c158d39a8efca67c04d4c70079deb61edc6497a04a50ec2\": container with ID starting with b1085efb676150742c158d39a8efca67c04d4c70079deb61edc6497a04a50ec2 not found: ID does not exist" containerID="b1085efb676150742c158d39a8efca67c04d4c70079deb61edc6497a04a50ec2"
Sep 09 03:37:37.488171 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.488112    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1085efb676150742c158d39a8efca67c04d4c70079deb61edc6497a04a50ec2"} err="failed to get container status \"b1085efb676150742c158d39a8efca67c04d4c70079deb61edc6497a04a50ec2\": rpc error: code = NotFound desc = could not find container \"b1085efb676150742c158d39a8efca67c04d4c70079deb61edc6497a04a50ec2\": container with ID starting with b1085efb676150742c158d39a8efca67c04d4c70079deb61edc6497a04a50ec2 not found: ID does not exist"
Sep 09 03:37:37.492433 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.492417    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/410f304a-1734-433d-9a40-ab92dd7c303f-openshift-service-ca\") pod \"410f304a-1734-433d-9a40-ab92dd7c303f\" (UID: \"410f304a-1734-433d-9a40-ab92dd7c303f\") "
Sep 09 03:37:37.492486 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.492452    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/410f304a-1734-433d-9a40-ab92dd7c303f-maas-api-tls\") pod \"410f304a-1734-433d-9a40-ab92dd7c303f\" (UID: \"410f304a-1734-433d-9a40-ab92dd7c303f\") "
Sep 09 03:37:37.492486 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.492483    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm7tn\" (UniqueName: \"kubernetes.io/projected/410f304a-1734-433d-9a40-ab92dd7c303f-kube-api-access-lm7tn\") pod \"410f304a-1734-433d-9a40-ab92dd7c303f\" (UID: \"410f304a-1734-433d-9a40-ab92dd7c303f\") "
Sep 09 03:37:37.492589 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.492507    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410f304a-1734-433d-9a40-ab92dd7c303f-metrics-certs\") pod \"410f304a-1734-433d-9a40-ab92dd7c303f\" (UID: \"410f304a-1734-433d-9a40-ab92dd7c303f\") "
Sep 09 03:37:37.492792 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.492768    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/410f304a-1734-433d-9a40-ab92dd7c303f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "410f304a-1734-433d-9a40-ab92dd7c303f" (UID: "410f304a-1734-433d-9a40-ab92dd7c303f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:37:37.493043 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.493022    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/410f304a-1734-433d-9a40-ab92dd7c303f-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:37:37.494563 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.494539    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/410f304a-1734-433d-9a40-ab92dd7c303f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "410f304a-1734-433d-9a40-ab92dd7c303f" (UID: "410f304a-1734-433d-9a40-ab92dd7c303f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:37:37.494681 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.494579    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/410f304a-1734-433d-9a40-ab92dd7c303f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "410f304a-1734-433d-9a40-ab92dd7c303f" (UID: "410f304a-1734-433d-9a40-ab92dd7c303f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:37:37.494681 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.494658    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/410f304a-1734-433d-9a40-ab92dd7c303f-kube-api-access-lm7tn" (OuterVolumeSpecName: "kube-api-access-lm7tn") pod "410f304a-1734-433d-9a40-ab92dd7c303f" (UID: "410f304a-1734-433d-9a40-ab92dd7c303f"). InnerVolumeSpecName "kube-api-access-lm7tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:37:37.594239 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.594199    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/410f304a-1734-433d-9a40-ab92dd7c303f-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:37:37.594239 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.594231    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm7tn\" (UniqueName: \"kubernetes.io/projected/410f304a-1734-433d-9a40-ab92dd7c303f-kube-api-access-lm7tn\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:37:37.594239 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.594240    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/410f304a-1734-433d-9a40-ab92dd7c303f-metrics-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:37:37.803278 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.803247    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"]
Sep 09 03:37:37.808929 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:37.808908    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-123302cc-848d4c67bc-7wmwp"]
Sep 09 03:37:38.116544 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:38.116501    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:38.198063 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:38.198028    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:39.359505 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:39.359472    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="410f304a-1734-433d-9a40-ab92dd7c303f" path="/var/lib/kubelet/pods/410f304a-1734-433d-9a40-ab92dd7c303f/volumes"
Sep 09 03:37:40.585311 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:40.585273    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:45.567386 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:45.567352    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"]
Sep 09 03:37:45.662374 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:45.662338    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:37:50.695237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:37:50.695200    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:38:00.588255 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:00.588196    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc" podUID="3f442ec8-955c-4e4e-abd3-de1f6906f528" containerName="main" containerID="cri-o://bf8d243765f6e57e7f5366747030f7564206d63f6feda308dd3626a7cfef2bce" gracePeriod=45
Sep 09 03:38:00.838559 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:00.838497    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:38:00.915534 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:00.915503    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-tmp-dir\") pod \"3f442ec8-955c-4e4e-abd3-de1f6906f528\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") "
Sep 09 03:38:00.915744 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:00.915558    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x72gp\" (UniqueName: \"kubernetes.io/projected/3f442ec8-955c-4e4e-abd3-de1f6906f528-kube-api-access-x72gp\") pod \"3f442ec8-955c-4e4e-abd3-de1f6906f528\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") "
Sep 09 03:38:00.915744 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:00.915647    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-home\") pod \"3f442ec8-955c-4e4e-abd3-de1f6906f528\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") "
Sep 09 03:38:00.915744 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:00.915679    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f442ec8-955c-4e4e-abd3-de1f6906f528-tls-certs\") pod \"3f442ec8-955c-4e4e-abd3-de1f6906f528\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") "
Sep 09 03:38:00.915744 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:00.915711    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-dshm\") pod \"3f442ec8-955c-4e4e-abd3-de1f6906f528\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") "
Sep 09 03:38:00.915744 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:00.915746    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-model-cache\") pod \"3f442ec8-955c-4e4e-abd3-de1f6906f528\" (UID: \"3f442ec8-955c-4e4e-abd3-de1f6906f528\") "
Sep 09 03:38:00.916016 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:00.915836    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f442ec8-955c-4e4e-abd3-de1f6906f528" (UID: "3f442ec8-955c-4e4e-abd3-de1f6906f528"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:38:00.916016 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:00.915940    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-home" (OuterVolumeSpecName: "home") pod "3f442ec8-955c-4e4e-abd3-de1f6906f528" (UID: "3f442ec8-955c-4e4e-abd3-de1f6906f528"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:38:00.916121 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:00.916048    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f442ec8-955c-4e4e-abd3-de1f6906f528" (UID: "3f442ec8-955c-4e4e-abd3-de1f6906f528"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:38:00.916187 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:00.916157    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-home\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:38:00.916187 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:00.916171    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-model-cache\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:38:00.916187 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:00.916182    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-tmp-dir\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:38:00.917873 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:00.917845    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f442ec8-955c-4e4e-abd3-de1f6906f528-kube-api-access-x72gp" (OuterVolumeSpecName: "kube-api-access-x72gp") pod "3f442ec8-955c-4e4e-abd3-de1f6906f528" (UID: "3f442ec8-955c-4e4e-abd3-de1f6906f528"). InnerVolumeSpecName "kube-api-access-x72gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:38:00.918014 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:00.917888    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f442ec8-955c-4e4e-abd3-de1f6906f528-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f442ec8-955c-4e4e-abd3-de1f6906f528" (UID: "3f442ec8-955c-4e4e-abd3-de1f6906f528"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:38:00.918094 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:00.918021    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-dshm" (OuterVolumeSpecName: "dshm") pod "3f442ec8-955c-4e4e-abd3-de1f6906f528" (UID: "3f442ec8-955c-4e4e-abd3-de1f6906f528"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:38:01.016959 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:01.016924    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f442ec8-955c-4e4e-abd3-de1f6906f528-tls-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:38:01.016959 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:01.016950    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f442ec8-955c-4e4e-abd3-de1f6906f528-dshm\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:38:01.016959 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:01.016960    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x72gp\" (UniqueName: \"kubernetes.io/projected/3f442ec8-955c-4e4e-abd3-de1f6906f528-kube-api-access-x72gp\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:38:01.620683 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:01.620587    2560 generic.go:356] "Generic (PLEG): container finished" podID="3f442ec8-955c-4e4e-abd3-de1f6906f528" containerID="bf8d243765f6e57e7f5366747030f7564206d63f6feda308dd3626a7cfef2bce" exitCode=0
Sep 09 03:38:01.620683 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:01.620668    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"
Sep 09 03:38:01.621091 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:01.620686    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc" event={"ID":"3f442ec8-955c-4e4e-abd3-de1f6906f528","Type":"ContainerDied","Data":"bf8d243765f6e57e7f5366747030f7564206d63f6feda308dd3626a7cfef2bce"}
Sep 09 03:38:01.621091 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:01.620730    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc" event={"ID":"3f442ec8-955c-4e4e-abd3-de1f6906f528","Type":"ContainerDied","Data":"07bffc880878987497df37985eccd20d7d8d71efec518f79eafdcfbef9368f0d"}
Sep 09 03:38:01.621091 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:01.620751    2560 scope.go:117] "RemoveContainer" containerID="bf8d243765f6e57e7f5366747030f7564206d63f6feda308dd3626a7cfef2bce"
Sep 09 03:38:01.629948 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:01.629923    2560 scope.go:117] "RemoveContainer" containerID="bf8d243765f6e57e7f5366747030f7564206d63f6feda308dd3626a7cfef2bce"
Sep 09 03:38:01.630254 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:38:01.630234    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf8d243765f6e57e7f5366747030f7564206d63f6feda308dd3626a7cfef2bce\": container with ID starting with bf8d243765f6e57e7f5366747030f7564206d63f6feda308dd3626a7cfef2bce not found: ID does not exist" containerID="bf8d243765f6e57e7f5366747030f7564206d63f6feda308dd3626a7cfef2bce"
Sep 09 03:38:01.630326 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:01.630264    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf8d243765f6e57e7f5366747030f7564206d63f6feda308dd3626a7cfef2bce"} err="failed to get container status \"bf8d243765f6e57e7f5366747030f7564206d63f6feda308dd3626a7cfef2bce\": rpc error: code = NotFound desc = could not find container \"bf8d243765f6e57e7f5366747030f7564206d63f6feda308dd3626a7cfef2bce\": container with ID starting with bf8d243765f6e57e7f5366747030f7564206d63f6feda308dd3626a7cfef2bce not found: ID does not exist"
Sep 09 03:38:01.640778 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:01.640752    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"]
Sep 09 03:38:01.642929 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:01.642905    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-123302cc/e2e-lifecycle-model-123302cc-kserve-8656cc7d95-d4fmc"]
Sep 09 03:38:03.367693 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:03.367652    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f442ec8-955c-4e4e-abd3-de1f6906f528" path="/var/lib/kubelet/pods/3f442ec8-955c-4e4e-abd3-de1f6906f528/volumes"
Sep 09 03:38:03.588440 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:03.588404    2560 scope.go:117] "RemoveContainer" containerID="b3eab3861ae34839e63f443684bb4faf403069f80c2d2175d1cda7cb63500b9c"
Sep 09 03:38:03.596698 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:03.596680    2560 scope.go:117] "RemoveContainer" containerID="04d3c55f7c155dd38c9dea5b1aacdcad8d107e6da4410a1b1ce6d97dc96ea39a"
Sep 09 03:38:06.887910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:06.887880    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:38:09.297349 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:09.297314    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7650-06895c"]
Sep 09 03:38:09.297933 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:09.297912    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f442ec8-955c-4e4e-abd3-de1f6906f528" containerName="main"
Sep 09 03:38:09.298018 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:09.297936    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f442ec8-955c-4e4e-abd3-de1f6906f528" containerName="main"
Sep 09 03:38:09.298018 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:09.297952    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="410f304a-1734-433d-9a40-ab92dd7c303f" containerName="maas-api"
Sep 09 03:38:09.298018 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:09.297960    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="410f304a-1734-433d-9a40-ab92dd7c303f" containerName="maas-api"
Sep 09 03:38:09.298173 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:09.298085    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="410f304a-1734-433d-9a40-ab92dd7c303f" containerName="maas-api"
Sep 09 03:38:09.298173 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:09.298101    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f442ec8-955c-4e4e-abd3-de1f6906f528" containerName="main"
Sep 09 03:38:09.301237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:09.301216    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7650-06895c"
Sep 09 03:38:09.304544 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:09.304525    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-k4rsk\""
Sep 09 03:38:09.323544 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:09.323520    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7650-06895c"]
Sep 09 03:38:09.390842 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:09.390804    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4ln5\" (UniqueName: \"kubernetes.io/projected/3646c366-dc8e-401f-b8f6-7627c96c1403-kube-api-access-n4ln5\") pod \"mt-curl-7650-06895c\" (UID: \"3646c366-dc8e-401f-b8f6-7627c96c1403\") " pod="opendatahub/mt-curl-7650-06895c"
Sep 09 03:38:09.471789 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:09.471764    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:38:09.491439 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:09.491409    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4ln5\" (UniqueName: \"kubernetes.io/projected/3646c366-dc8e-401f-b8f6-7627c96c1403-kube-api-access-n4ln5\") pod \"mt-curl-7650-06895c\" (UID: \"3646c366-dc8e-401f-b8f6-7627c96c1403\") " pod="opendatahub/mt-curl-7650-06895c"
Sep 09 03:38:09.501663 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:09.501638    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:38:09.504709 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:09.504684    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4ln5\" (UniqueName: \"kubernetes.io/projected/3646c366-dc8e-401f-b8f6-7627c96c1403-kube-api-access-n4ln5\") pod \"mt-curl-7650-06895c\" (UID: \"3646c366-dc8e-401f-b8f6-7627c96c1403\") " pod="opendatahub/mt-curl-7650-06895c"
Sep 09 03:38:09.609366 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:09.609343    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7650-06895c"
Sep 09 03:38:09.883387 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:09.883360    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7650-06895c"]
Sep 09 03:38:09.885658 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:38:09.885624    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3646c366_dc8e_401f_b8f6_7627c96c1403.slice/crio-6e9fbb1ace2cbab9f65e2c2391be333b3e4315adfc78e0fcc62649240aff1f38 WatchSource:0}: Error finding container 6e9fbb1ace2cbab9f65e2c2391be333b3e4315adfc78e0fcc62649240aff1f38: Status 404 returned error can't find the container with id 6e9fbb1ace2cbab9f65e2c2391be333b3e4315adfc78e0fcc62649240aff1f38
Sep 09 03:38:10.678913 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:10.678879    2560 generic.go:356] "Generic (PLEG): container finished" podID="3646c366-dc8e-401f-b8f6-7627c96c1403" containerID="8a5eed84b5d130782b5f30aaed56117dc74ae4893066d0b4b57117625bb6c027" exitCode=0
Sep 09 03:38:10.679219 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:10.678958    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7650-06895c" event={"ID":"3646c366-dc8e-401f-b8f6-7627c96c1403","Type":"ContainerDied","Data":"8a5eed84b5d130782b5f30aaed56117dc74ae4893066d0b4b57117625bb6c027"}
Sep 09 03:38:10.679219 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:10.678995    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7650-06895c" event={"ID":"3646c366-dc8e-401f-b8f6-7627c96c1403","Type":"ContainerStarted","Data":"6e9fbb1ace2cbab9f65e2c2391be333b3e4315adfc78e0fcc62649240aff1f38"}
Sep 09 03:38:11.819972 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:11.819942    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7650-06895c"
Sep 09 03:38:11.840026 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:11.840002    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7650-06895c_3646c366-dc8e-401f-b8f6-7627c96c1403/mt-curl-7650-06895c/0.log"
Sep 09 03:38:11.861903 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:11.861873    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7650-06895c"]
Sep 09 03:38:11.864375 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:11.864354    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7650-06895c"]
Sep 09 03:38:11.913993 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:11.913966    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4ln5\" (UniqueName: \"kubernetes.io/projected/3646c366-dc8e-401f-b8f6-7627c96c1403-kube-api-access-n4ln5\") pod \"3646c366-dc8e-401f-b8f6-7627c96c1403\" (UID: \"3646c366-dc8e-401f-b8f6-7627c96c1403\") "
Sep 09 03:38:11.915991 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:11.915967    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3646c366-dc8e-401f-b8f6-7627c96c1403-kube-api-access-n4ln5" (OuterVolumeSpecName: "kube-api-access-n4ln5") pod "3646c366-dc8e-401f-b8f6-7627c96c1403" (UID: "3646c366-dc8e-401f-b8f6-7627c96c1403"). InnerVolumeSpecName "kube-api-access-n4ln5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:38:12.015751 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:12.015681    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4ln5\" (UniqueName: \"kubernetes.io/projected/3646c366-dc8e-401f-b8f6-7627c96c1403-kube-api-access-n4ln5\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:38:12.689886 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:12.689859    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e9fbb1ace2cbab9f65e2c2391be333b3e4315adfc78e0fcc62649240aff1f38"
Sep 09 03:38:12.689886 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:12.689883    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7650-06895c"
Sep 09 03:38:12.700526 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:38:12.700498    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7650-06895c\" is forbidden: User \"system:node:ip-10-0-133-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-131.ec2.internal' and this object" podUID="3646c366-dc8e-401f-b8f6-7627c96c1403" pod="opendatahub/mt-curl-7650-06895c"
Sep 09 03:38:13.316560 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:13.316511    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:13.357763 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:38:13.357728    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7650-06895c\" is forbidden: User \"system:node:ip-10-0-133-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-131.ec2.internal' and this object" podUID="3646c366-dc8e-401f-b8f6-7627c96c1403" pod="opendatahub/mt-curl-7650-06895c"
Sep 09 03:38:13.358880 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:13.358853    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3646c366-dc8e-401f-b8f6-7627c96c1403" path="/var/lib/kubelet/pods/3646c366-dc8e-401f-b8f6-7627c96c1403/volumes"
Sep 09 03:38:14.316934 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:14.316888    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:18.400189 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:18.400156    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:38:29.079585 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:29.079504    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:29.564251 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:29.564217    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:35.365515 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:35.365482    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:35.468293 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:35.468241    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:39.693683 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:39.693644    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:38:41.621413 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:41.621379    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:41.723410 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:41.723369    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:41.937056 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:41.936970    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:43.448681 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:43.448638    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:43.552683 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:43.552656    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:43.648835 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:43.648803    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:44.390097 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:44.390061    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:44.480151 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:44.480121    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:44.585983 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:44.585945    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:45.979509 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:45.979476    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:50.869628 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:50.869574    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:51.822837 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:51.822803    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:51.922622 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:51.922576    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:52.593260 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:52.593224    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:58.621050 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:58.621002    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:58.717484 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:58.717454    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:38:58.814693 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:38:58.814650    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:09.447460 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:09.447425    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:10.581745 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:10.581708    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:11.475720 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:11.475687    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:11.941986 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:11.941944    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:15.896496 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:15.896458    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:39:19.064059 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:19.064020    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:19.141399 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:19.141366    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:30.759160 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:30.759121    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:36.825019 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:36.824983    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:36.925455 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:36.925421    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:37.278054 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:37.278022    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:37.589687 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:37.589583    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:37.811554 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:37.811507    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:41.201576 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:41.201538    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:41.331077 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:41.331044    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:42.305113 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.305024    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"]
Sep 09 03:39:42.305710 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.305689    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3646c366-dc8e-401f-b8f6-7627c96c1403" containerName="mt-curl-7650-06895c"
Sep 09 03:39:42.305763 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.305714    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3646c366-dc8e-401f-b8f6-7627c96c1403" containerName="mt-curl-7650-06895c"
Sep 09 03:39:42.305854 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.305842    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3646c366-dc8e-401f-b8f6-7627c96c1403" containerName="mt-curl-7650-06895c"
Sep 09 03:39:42.309208 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.309187    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:39:42.309442 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.309424    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:39:42.311813 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.311794    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7f0891-serving-cert\""
Sep 09 03:39:42.319860 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.319841    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"]
Sep 09 03:39:42.339999 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.339961    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:42.405020 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.404990    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3419d76e-0623-4f33-9c60-4ea7386fdbca-metrics-certs\") pod \"maas-api-e2e-api-a-7f0891-845c88967d-sfd8g\" (UID: \"3419d76e-0623-4f33-9c60-4ea7386fdbca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:39:42.405166 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.405121    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3419d76e-0623-4f33-9c60-4ea7386fdbca-openshift-service-ca\") pod \"maas-api-e2e-api-a-7f0891-845c88967d-sfd8g\" (UID: \"3419d76e-0623-4f33-9c60-4ea7386fdbca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:39:42.405231 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.405206    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmrzv\" (UniqueName: \"kubernetes.io/projected/3419d76e-0623-4f33-9c60-4ea7386fdbca-kube-api-access-lmrzv\") pod \"maas-api-e2e-api-a-7f0891-845c88967d-sfd8g\" (UID: \"3419d76e-0623-4f33-9c60-4ea7386fdbca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:39:42.405289 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.405249    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3419d76e-0623-4f33-9c60-4ea7386fdbca-maas-api-tls\") pod \"maas-api-e2e-api-a-7f0891-845c88967d-sfd8g\" (UID: \"3419d76e-0623-4f33-9c60-4ea7386fdbca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:39:42.441529 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.441502    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:42.506410 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.506382    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3419d76e-0623-4f33-9c60-4ea7386fdbca-openshift-service-ca\") pod \"maas-api-e2e-api-a-7f0891-845c88967d-sfd8g\" (UID: \"3419d76e-0623-4f33-9c60-4ea7386fdbca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:39:42.506559 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.506431    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmrzv\" (UniqueName: \"kubernetes.io/projected/3419d76e-0623-4f33-9c60-4ea7386fdbca-kube-api-access-lmrzv\") pod \"maas-api-e2e-api-a-7f0891-845c88967d-sfd8g\" (UID: \"3419d76e-0623-4f33-9c60-4ea7386fdbca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:39:42.506559 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.506458    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3419d76e-0623-4f33-9c60-4ea7386fdbca-maas-api-tls\") pod \"maas-api-e2e-api-a-7f0891-845c88967d-sfd8g\" (UID: \"3419d76e-0623-4f33-9c60-4ea7386fdbca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:39:42.506559 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.506504    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3419d76e-0623-4f33-9c60-4ea7386fdbca-metrics-certs\") pod \"maas-api-e2e-api-a-7f0891-845c88967d-sfd8g\" (UID: \"3419d76e-0623-4f33-9c60-4ea7386fdbca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:39:42.506766 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:39:42.506642    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-serving-cert: secret "maas-api-e2e-api-a-7f0891-serving-cert" not found
Sep 09 03:39:42.506766 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:39:42.506707    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3419d76e-0623-4f33-9c60-4ea7386fdbca-maas-api-tls podName:3419d76e-0623-4f33-9c60-4ea7386fdbca nodeName:}" failed. No retries permitted until 2026-09-09 03:39:43.006688449 +0000 UTC m=+1600.143436300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3419d76e-0623-4f33-9c60-4ea7386fdbca-maas-api-tls") pod "maas-api-e2e-api-a-7f0891-845c88967d-sfd8g" (UID: "3419d76e-0623-4f33-9c60-4ea7386fdbca") : secret "maas-api-e2e-api-a-7f0891-serving-cert" not found
Sep 09 03:39:42.507143 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.507118    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3419d76e-0623-4f33-9c60-4ea7386fdbca-openshift-service-ca\") pod \"maas-api-e2e-api-a-7f0891-845c88967d-sfd8g\" (UID: \"3419d76e-0623-4f33-9c60-4ea7386fdbca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:39:42.508955 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.508938    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3419d76e-0623-4f33-9c60-4ea7386fdbca-metrics-certs\") pod \"maas-api-e2e-api-a-7f0891-845c88967d-sfd8g\" (UID: \"3419d76e-0623-4f33-9c60-4ea7386fdbca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:39:42.517226 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.517202    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmrzv\" (UniqueName: \"kubernetes.io/projected/3419d76e-0623-4f33-9c60-4ea7386fdbca-kube-api-access-lmrzv\") pod \"maas-api-e2e-api-a-7f0891-845c88967d-sfd8g\" (UID: \"3419d76e-0623-4f33-9c60-4ea7386fdbca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:39:42.617437 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.617404    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:42.893375 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:42.893272    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:43.011229 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:43.011194    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3419d76e-0623-4f33-9c60-4ea7386fdbca-maas-api-tls\") pod \"maas-api-e2e-api-a-7f0891-845c88967d-sfd8g\" (UID: \"3419d76e-0623-4f33-9c60-4ea7386fdbca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:39:43.013524 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:43.013493    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:43.014080 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:43.014060    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3419d76e-0623-4f33-9c60-4ea7386fdbca-maas-api-tls\") pod \"maas-api-e2e-api-a-7f0891-845c88967d-sfd8g\" (UID: \"3419d76e-0623-4f33-9c60-4ea7386fdbca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:39:43.220667 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:43.220564    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:39:43.494059 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:43.494034    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"]
Sep 09 03:39:43.496066 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:39:43.496042    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3419d76e_0623_4f33_9c60_4ea7386fdbca.slice/crio-9d7e408515fed650c0f31f15d7724c44a4415c808cdf23c78776d1b974afc774 WatchSource:0}: Error finding container 9d7e408515fed650c0f31f15d7724c44a4415c808cdf23c78776d1b974afc774: Status 404 returned error can't find the container with id 9d7e408515fed650c0f31f15d7724c44a4415c808cdf23c78776d1b974afc774
Sep 09 03:39:43.497415 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:43.497398    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:39:44.155556 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:44.155462    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g" event={"ID":"3419d76e-0623-4f33-9c60-4ea7386fdbca","Type":"ContainerStarted","Data":"4272bf040b381f977bbbf057d2bc21d230eb3f1120cdb99e1ed13d33c72efb0b"}
Sep 09 03:39:44.155735 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:44.155554    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g" event={"ID":"3419d76e-0623-4f33-9c60-4ea7386fdbca","Type":"ContainerStarted","Data":"9d7e408515fed650c0f31f15d7724c44a4415c808cdf23c78776d1b974afc774"}
Sep 09 03:39:44.155735 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:44.155586    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:39:44.178051 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:44.178004    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g" podStartSLOduration=1.8083803189999998 podStartE2EDuration="2.177988706s" totalImagesPullingTime="369.608387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:39:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:39:44.17633236 +0000 UTC m=+1601.313080227" watchObservedRunningTime="2026-09-09 03:39:44.177988706 +0000 UTC m=+1601.314736571"
Sep 09 03:39:47.704664 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:47.704618    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:39:50.165619 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:50.165578    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:39:56.282461 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:56.282357    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7650-f6c010"]
Sep 09 03:39:56.286177 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:56.286161    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7650-f6c010"
Sep 09 03:39:56.289475 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:56.289454    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-k4rsk\""
Sep 09 03:39:56.291303 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:56.291280    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7650-f6c010"]
Sep 09 03:39:56.343461 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:56.343430    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwtd9\" (UniqueName: \"kubernetes.io/projected/156a1bc5-1ce8-4337-905c-b4445dc5b245-kube-api-access-fwtd9\") pod \"mt-curl-7650-f6c010\" (UID: \"156a1bc5-1ce8-4337-905c-b4445dc5b245\") " pod="opendatahub/mt-curl-7650-f6c010"
Sep 09 03:39:56.444520 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:56.444482    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwtd9\" (UniqueName: \"kubernetes.io/projected/156a1bc5-1ce8-4337-905c-b4445dc5b245-kube-api-access-fwtd9\") pod \"mt-curl-7650-f6c010\" (UID: \"156a1bc5-1ce8-4337-905c-b4445dc5b245\") " pod="opendatahub/mt-curl-7650-f6c010"
Sep 09 03:39:56.454397 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:56.454365    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwtd9\" (UniqueName: \"kubernetes.io/projected/156a1bc5-1ce8-4337-905c-b4445dc5b245-kube-api-access-fwtd9\") pod \"mt-curl-7650-f6c010\" (UID: \"156a1bc5-1ce8-4337-905c-b4445dc5b245\") " pod="opendatahub/mt-curl-7650-f6c010"
Sep 09 03:39:56.598175 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:56.598068    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7650-f6c010"
Sep 09 03:39:56.868732 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:56.868708    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7650-f6c010"]
Sep 09 03:39:56.870620 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:39:56.870580    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod156a1bc5_1ce8_4337_905c_b4445dc5b245.slice/crio-e03b807d908b288fceb0908404e7cc6f510f506650348ff20736eae0722e6b3e WatchSource:0}: Error finding container e03b807d908b288fceb0908404e7cc6f510f506650348ff20736eae0722e6b3e: Status 404 returned error can't find the container with id e03b807d908b288fceb0908404e7cc6f510f506650348ff20736eae0722e6b3e
Sep 09 03:39:57.236566 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:57.236528    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7650-f6c010" event={"ID":"156a1bc5-1ce8-4337-905c-b4445dc5b245","Type":"ContainerStarted","Data":"1ca6b389b8ec7886e08118b10e2d3a0a8b2c28f6bb5cde058e0cd4f1e8c17203"}
Sep 09 03:39:57.236709 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:57.236582    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7650-f6c010" event={"ID":"156a1bc5-1ce8-4337-905c-b4445dc5b245","Type":"ContainerStarted","Data":"e03b807d908b288fceb0908404e7cc6f510f506650348ff20736eae0722e6b3e"}
Sep 09 03:39:57.254768 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:57.254726    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/mt-curl-7650-f6c010" podStartSLOduration=0.988374332 podStartE2EDuration="1.254710332s" totalImagesPullingTime="266.336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:39:57.252943143 +0000 UTC m=+1614.389691008" watchObservedRunningTime="2026-09-09 03:39:57.254710332 +0000 UTC m=+1614.391458208"
Sep 09 03:39:57.349355 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:57.349329    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7650-f6c010_156a1bc5-1ce8-4337-905c-b4445dc5b245/mt-curl-7650-f6c010/0.log"
Sep 09 03:39:58.242549 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:58.242513    2560 generic.go:356] "Generic (PLEG): container finished" podID="156a1bc5-1ce8-4337-905c-b4445dc5b245" containerID="1ca6b389b8ec7886e08118b10e2d3a0a8b2c28f6bb5cde058e0cd4f1e8c17203" exitCode=0
Sep 09 03:39:58.242749 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:58.242618    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7650-f6c010" event={"ID":"156a1bc5-1ce8-4337-905c-b4445dc5b245","Type":"ContainerDied","Data":"1ca6b389b8ec7886e08118b10e2d3a0a8b2c28f6bb5cde058e0cd4f1e8c17203"}
Sep 09 03:39:59.380346 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:59.380323    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7650-f6c010"
Sep 09 03:39:59.416826 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:59.416796    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7650-f6c010"]
Sep 09 03:39:59.420488 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:59.420460    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7650-f6c010"]
Sep 09 03:39:59.468954 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:59.468927    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwtd9\" (UniqueName: \"kubernetes.io/projected/156a1bc5-1ce8-4337-905c-b4445dc5b245-kube-api-access-fwtd9\") pod \"156a1bc5-1ce8-4337-905c-b4445dc5b245\" (UID: \"156a1bc5-1ce8-4337-905c-b4445dc5b245\") "
Sep 09 03:39:59.470969 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:59.470948    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156a1bc5-1ce8-4337-905c-b4445dc5b245-kube-api-access-fwtd9" (OuterVolumeSpecName: "kube-api-access-fwtd9") pod "156a1bc5-1ce8-4337-905c-b4445dc5b245" (UID: "156a1bc5-1ce8-4337-905c-b4445dc5b245"). InnerVolumeSpecName "kube-api-access-fwtd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:39:59.552486 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:59.552413    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7650-8dbb4b"]
Sep 09 03:39:59.552845 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:59.552832    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="156a1bc5-1ce8-4337-905c-b4445dc5b245" containerName="mt-curl-7650-f6c010"
Sep 09 03:39:59.552887 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:59.552847    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="156a1bc5-1ce8-4337-905c-b4445dc5b245" containerName="mt-curl-7650-f6c010"
Sep 09 03:39:59.552934 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:59.552920    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="156a1bc5-1ce8-4337-905c-b4445dc5b245" containerName="mt-curl-7650-f6c010"
Sep 09 03:39:59.556026 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:59.556011    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7650-8dbb4b"
Sep 09 03:39:59.563444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:59.563419    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7650-8dbb4b"]
Sep 09 03:39:59.570225 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:59.570198    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnvz5\" (UniqueName: \"kubernetes.io/projected/d4cdd8ca-53b6-4964-9022-b648fb9f4140-kube-api-access-hnvz5\") pod \"mt-curl-7650-8dbb4b\" (UID: \"d4cdd8ca-53b6-4964-9022-b648fb9f4140\") " pod="opendatahub/mt-curl-7650-8dbb4b"
Sep 09 03:39:59.570369 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:59.570264    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwtd9\" (UniqueName: \"kubernetes.io/projected/156a1bc5-1ce8-4337-905c-b4445dc5b245-kube-api-access-fwtd9\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:39:59.671476 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:59.671440    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnvz5\" (UniqueName: \"kubernetes.io/projected/d4cdd8ca-53b6-4964-9022-b648fb9f4140-kube-api-access-hnvz5\") pod \"mt-curl-7650-8dbb4b\" (UID: \"d4cdd8ca-53b6-4964-9022-b648fb9f4140\") " pod="opendatahub/mt-curl-7650-8dbb4b"
Sep 09 03:39:59.680183 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:59.680152    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnvz5\" (UniqueName: \"kubernetes.io/projected/d4cdd8ca-53b6-4964-9022-b648fb9f4140-kube-api-access-hnvz5\") pod \"mt-curl-7650-8dbb4b\" (UID: \"d4cdd8ca-53b6-4964-9022-b648fb9f4140\") " pod="opendatahub/mt-curl-7650-8dbb4b"
Sep 09 03:39:59.866570 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:39:59.866542    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7650-8dbb4b"
Sep 09 03:40:00.139840 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:00.139816    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7650-8dbb4b"]
Sep 09 03:40:00.141224 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:40:00.141195    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4cdd8ca_53b6_4964_9022_b648fb9f4140.slice/crio-62a8bb3bc433f0d302463150d1b7a2052aae6c69edde66755c59fd18a1fe75cf WatchSource:0}: Error finding container 62a8bb3bc433f0d302463150d1b7a2052aae6c69edde66755c59fd18a1fe75cf: Status 404 returned error can't find the container with id 62a8bb3bc433f0d302463150d1b7a2052aae6c69edde66755c59fd18a1fe75cf
Sep 09 03:40:00.254131 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:00.254104    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7650-f6c010"
Sep 09 03:40:00.254307 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:00.254134    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e03b807d908b288fceb0908404e7cc6f510f506650348ff20736eae0722e6b3e"
Sep 09 03:40:00.255330 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:00.255305    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7650-8dbb4b" event={"ID":"d4cdd8ca-53b6-4964-9022-b648fb9f4140","Type":"ContainerStarted","Data":"62a8bb3bc433f0d302463150d1b7a2052aae6c69edde66755c59fd18a1fe75cf"}
Sep 09 03:40:00.265288 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:40:00.265262    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7650-f6c010\" is forbidden: User \"system:node:ip-10-0-133-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-131.ec2.internal' and this object" podUID="156a1bc5-1ce8-4337-905c-b4445dc5b245" pod="opendatahub/mt-curl-7650-f6c010"
Sep 09 03:40:01.262112 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:01.262077    2560 generic.go:356] "Generic (PLEG): container finished" podID="d4cdd8ca-53b6-4964-9022-b648fb9f4140" containerID="96306c8f0672e8c8479d6e71f2c2220f80aef9c38c457059707cd33d60ad6910" exitCode=0
Sep 09 03:40:01.262501 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:01.262163    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7650-8dbb4b" event={"ID":"d4cdd8ca-53b6-4964-9022-b648fb9f4140","Type":"ContainerDied","Data":"96306c8f0672e8c8479d6e71f2c2220f80aef9c38c457059707cd33d60ad6910"}
Sep 09 03:40:01.264625 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:40:01.264579    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7650-f6c010\" is forbidden: User \"system:node:ip-10-0-133-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-131.ec2.internal' and this object" podUID="156a1bc5-1ce8-4337-905c-b4445dc5b245" pod="opendatahub/mt-curl-7650-f6c010"
Sep 09 03:40:01.358572 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:01.358539    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="156a1bc5-1ce8-4337-905c-b4445dc5b245" path="/var/lib/kubelet/pods/156a1bc5-1ce8-4337-905c-b4445dc5b245/volumes"
Sep 09 03:40:01.887848 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:01.887813    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:02.405760 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:02.405730    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7650-8dbb4b"
Sep 09 03:40:02.429003 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:02.428962    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7650-8dbb4b_d4cdd8ca-53b6-4964-9022-b648fb9f4140/mt-curl-7650-8dbb4b/0.log"
Sep 09 03:40:02.451136 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:02.451098    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7650-8dbb4b"]
Sep 09 03:40:02.457248 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:02.457214    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7650-8dbb4b"]
Sep 09 03:40:02.497694 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:02.497662    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnvz5\" (UniqueName: \"kubernetes.io/projected/d4cdd8ca-53b6-4964-9022-b648fb9f4140-kube-api-access-hnvz5\") pod \"d4cdd8ca-53b6-4964-9022-b648fb9f4140\" (UID: \"d4cdd8ca-53b6-4964-9022-b648fb9f4140\") "
Sep 09 03:40:02.499762 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:02.499729    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4cdd8ca-53b6-4964-9022-b648fb9f4140-kube-api-access-hnvz5" (OuterVolumeSpecName: "kube-api-access-hnvz5") pod "d4cdd8ca-53b6-4964-9022-b648fb9f4140" (UID: "d4cdd8ca-53b6-4964-9022-b648fb9f4140"). InnerVolumeSpecName "kube-api-access-hnvz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:40:02.598705 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:02.598570    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnvz5\" (UniqueName: \"kubernetes.io/projected/d4cdd8ca-53b6-4964-9022-b648fb9f4140-kube-api-access-hnvz5\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:40:03.275612 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:03.275567    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62a8bb3bc433f0d302463150d1b7a2052aae6c69edde66755c59fd18a1fe75cf"
Sep 09 03:40:03.275766 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:03.275616    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7650-8dbb4b"
Sep 09 03:40:03.286002 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:40:03.285965    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7650-8dbb4b\" is forbidden: User \"system:node:ip-10-0-133-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-131.ec2.internal' and this object" podUID="d4cdd8ca-53b6-4964-9022-b648fb9f4140" pod="opendatahub/mt-curl-7650-8dbb4b"
Sep 09 03:40:03.332505 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:03.332471    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:03.357448 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:40:03.357406    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7650-8dbb4b\" is forbidden: User \"system:node:ip-10-0-133-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-131.ec2.internal' and this object" podUID="d4cdd8ca-53b6-4964-9022-b648fb9f4140" pod="opendatahub/mt-curl-7650-8dbb4b"
Sep 09 03:40:03.359848 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:03.359826    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4cdd8ca-53b6-4964-9022-b648fb9f4140" path="/var/lib/kubelet/pods/d4cdd8ca-53b6-4964-9022-b648fb9f4140/volumes"
Sep 09 03:40:04.727461 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:04.727405    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:05.110731 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:05.110693    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:40:06.229038 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:06.229006    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:06.309450 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:06.309417    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:06.734235 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:06.734202    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:07.100864 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:07.100768    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:07.273570 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:07.273535    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:10.391514 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:10.391482    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:10.481537 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:10.481498    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:11.528353 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.528320    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"]
Sep 09 03:40:11.528950 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.528930    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4cdd8ca-53b6-4964-9022-b648fb9f4140" containerName="mt-curl-7650-8dbb4b"
Sep 09 03:40:11.528950 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.528950    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4cdd8ca-53b6-4964-9022-b648fb9f4140" containerName="mt-curl-7650-8dbb4b"
Sep 09 03:40:11.529117 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.529082    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4cdd8ca-53b6-4964-9022-b648fb9f4140" containerName="mt-curl-7650-8dbb4b"
Sep 09 03:40:11.532644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.532625    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:40:11.535692 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.535673    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1c4cb1-serving-cert\""
Sep 09 03:40:11.547625 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.547585    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"]
Sep 09 03:40:11.578430 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.578400    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:11.590798 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.590769    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dghcq\" (UniqueName: \"kubernetes.io/projected/cbe82c0b-8207-4344-96db-20d64785238d-kube-api-access-dghcq\") pod \"maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98\" (UID: \"cbe82c0b-8207-4344-96db-20d64785238d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:40:11.590920 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.590810    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe82c0b-8207-4344-96db-20d64785238d-metrics-certs\") pod \"maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98\" (UID: \"cbe82c0b-8207-4344-96db-20d64785238d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:40:11.590920 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.590873    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe82c0b-8207-4344-96db-20d64785238d-openshift-service-ca\") pod \"maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98\" (UID: \"cbe82c0b-8207-4344-96db-20d64785238d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:40:11.590920 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.590894    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbe82c0b-8207-4344-96db-20d64785238d-maas-api-tls\") pod \"maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98\" (UID: \"cbe82c0b-8207-4344-96db-20d64785238d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:40:11.681302 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.681261    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:11.692383 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.692353    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dghcq\" (UniqueName: \"kubernetes.io/projected/cbe82c0b-8207-4344-96db-20d64785238d-kube-api-access-dghcq\") pod \"maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98\" (UID: \"cbe82c0b-8207-4344-96db-20d64785238d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:40:11.692514 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.692390    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe82c0b-8207-4344-96db-20d64785238d-metrics-certs\") pod \"maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98\" (UID: \"cbe82c0b-8207-4344-96db-20d64785238d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:40:11.692514 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.692421    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe82c0b-8207-4344-96db-20d64785238d-openshift-service-ca\") pod \"maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98\" (UID: \"cbe82c0b-8207-4344-96db-20d64785238d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:40:11.692514 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.692443    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbe82c0b-8207-4344-96db-20d64785238d-maas-api-tls\") pod \"maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98\" (UID: \"cbe82c0b-8207-4344-96db-20d64785238d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:40:11.692677 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:40:11.692560    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-serving-cert: secret "maas-api-e2e-api-b-1c4cb1-serving-cert" not found
Sep 09 03:40:11.692677 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:40:11.692633    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe82c0b-8207-4344-96db-20d64785238d-maas-api-tls podName:cbe82c0b-8207-4344-96db-20d64785238d nodeName:}" failed. No retries permitted until 2026-09-09 03:40:12.192616846 +0000 UTC m=+1629.329364702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cbe82c0b-8207-4344-96db-20d64785238d-maas-api-tls") pod "maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98" (UID: "cbe82c0b-8207-4344-96db-20d64785238d") : secret "maas-api-e2e-api-b-1c4cb1-serving-cert" not found
Sep 09 03:40:11.693194 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.693172    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe82c0b-8207-4344-96db-20d64785238d-openshift-service-ca\") pod \"maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98\" (UID: \"cbe82c0b-8207-4344-96db-20d64785238d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:40:11.694942 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.694920    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe82c0b-8207-4344-96db-20d64785238d-metrics-certs\") pod \"maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98\" (UID: \"cbe82c0b-8207-4344-96db-20d64785238d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:40:11.706478 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.706450    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dghcq\" (UniqueName: \"kubernetes.io/projected/cbe82c0b-8207-4344-96db-20d64785238d-kube-api-access-dghcq\") pod \"maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98\" (UID: \"cbe82c0b-8207-4344-96db-20d64785238d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:40:11.949332 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:11.949295    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:12.104626 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:12.104580    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:12.196877 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:12.196839    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbe82c0b-8207-4344-96db-20d64785238d-maas-api-tls\") pod \"maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98\" (UID: \"cbe82c0b-8207-4344-96db-20d64785238d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:40:12.199252 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:12.199233    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbe82c0b-8207-4344-96db-20d64785238d-maas-api-tls\") pod \"maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98\" (UID: \"cbe82c0b-8207-4344-96db-20d64785238d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:40:12.201630 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:12.201544    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:12.443329 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:12.443299    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:40:12.720817 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:12.720792    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"]
Sep 09 03:40:12.722717 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:40:12.722693    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe82c0b_8207_4344_96db_20d64785238d.slice/crio-3262309c9cd98d02372bddd92c3bd8fd7dd8d64c25a49a645671c492ad66f70e WatchSource:0}: Error finding container 3262309c9cd98d02372bddd92c3bd8fd7dd8d64c25a49a645671c492ad66f70e: Status 404 returned error can't find the container with id 3262309c9cd98d02372bddd92c3bd8fd7dd8d64c25a49a645671c492ad66f70e
Sep 09 03:40:13.330380 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:13.330299    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98" event={"ID":"cbe82c0b-8207-4344-96db-20d64785238d","Type":"ContainerStarted","Data":"4e4e38047f64cc022053185b4012baa709208bcbd6ae974c7f3336719ec8a1e3"}
Sep 09 03:40:13.330380 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:13.330350    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98" event={"ID":"cbe82c0b-8207-4344-96db-20d64785238d","Type":"ContainerStarted","Data":"3262309c9cd98d02372bddd92c3bd8fd7dd8d64c25a49a645671c492ad66f70e"}
Sep 09 03:40:13.330567 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:13.330391    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:40:13.352198 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:13.352150    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98" podStartSLOduration=2.037705161 podStartE2EDuration="2.352135235s" totalImagesPullingTime="314.430074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:40:13.350322294 +0000 UTC m=+1630.487070158" watchObservedRunningTime="2026-09-09 03:40:13.352135235 +0000 UTC m=+1630.488883124"
Sep 09 03:40:19.341288 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:19.341259    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:40:19.899665 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:19.899634    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"]
Sep 09 03:40:19.904705 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:19.904686    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:19.907519 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:19.907496    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-5d8e85\"/\"openshift-service-ca.crt\""
Sep 09 03:40:19.908704 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:19.908684    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-5d8e85\"/\"default-dockercfg-r99j8\""
Sep 09 03:40:19.908824 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:19.908718    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-5d8e85\"/\"rate-test-model-5d8e85-kserve-self-signed-certs\""
Sep 09 03:40:19.908824 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:19.908743    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-5d8e85\"/\"kube-root-ca.crt\""
Sep 09 03:40:19.914327 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:19.914303    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"]
Sep 09 03:40:19.977083 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:19.977053    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-dshm\") pod \"rate-test-model-5d8e85-kserve-799488d95-v7tlm\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") " pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:19.977234 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:19.977116    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wj8k\" (UniqueName: \"kubernetes.io/projected/49773d57-b97f-40dc-b214-d2480b8990f1-kube-api-access-9wj8k\") pod \"rate-test-model-5d8e85-kserve-799488d95-v7tlm\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") " pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:19.977234 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:19.977150    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49773d57-b97f-40dc-b214-d2480b8990f1-tls-certs\") pod \"rate-test-model-5d8e85-kserve-799488d95-v7tlm\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") " pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:19.977234 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:19.977169    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-home\") pod \"rate-test-model-5d8e85-kserve-799488d95-v7tlm\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") " pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:19.977234 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:19.977182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-model-cache\") pod \"rate-test-model-5d8e85-kserve-799488d95-v7tlm\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") " pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:19.977367 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:19.977253    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-tmp-dir\") pod \"rate-test-model-5d8e85-kserve-799488d95-v7tlm\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") " pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:19.992015 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:19.991992    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:20.078636 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:20.078579    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wj8k\" (UniqueName: \"kubernetes.io/projected/49773d57-b97f-40dc-b214-d2480b8990f1-kube-api-access-9wj8k\") pod \"rate-test-model-5d8e85-kserve-799488d95-v7tlm\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") " pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:20.078802 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:20.078647    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49773d57-b97f-40dc-b214-d2480b8990f1-tls-certs\") pod \"rate-test-model-5d8e85-kserve-799488d95-v7tlm\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") " pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:20.078802 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:20.078676    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-home\") pod \"rate-test-model-5d8e85-kserve-799488d95-v7tlm\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") " pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:20.078802 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:20.078699    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-model-cache\") pod \"rate-test-model-5d8e85-kserve-799488d95-v7tlm\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") " pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:20.078802 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:20.078757    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-tmp-dir\") pod \"rate-test-model-5d8e85-kserve-799488d95-v7tlm\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") " pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:20.079008 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:20.078822    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-dshm\") pod \"rate-test-model-5d8e85-kserve-799488d95-v7tlm\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") " pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:20.079204 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:20.079181    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-model-cache\") pod \"rate-test-model-5d8e85-kserve-799488d95-v7tlm\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") " pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:20.079552 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:20.079144    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-home\") pod \"rate-test-model-5d8e85-kserve-799488d95-v7tlm\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") " pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:20.079691 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:20.079165    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-tmp-dir\") pod \"rate-test-model-5d8e85-kserve-799488d95-v7tlm\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") " pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:20.082016 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:20.081982    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-dshm\") pod \"rate-test-model-5d8e85-kserve-799488d95-v7tlm\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") " pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:20.082260 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:20.082235    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49773d57-b97f-40dc-b214-d2480b8990f1-tls-certs\") pod \"rate-test-model-5d8e85-kserve-799488d95-v7tlm\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") " pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:20.089020 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:20.088995    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wj8k\" (UniqueName: \"kubernetes.io/projected/49773d57-b97f-40dc-b214-d2480b8990f1-kube-api-access-9wj8k\") pod \"rate-test-model-5d8e85-kserve-799488d95-v7tlm\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") " pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:20.215283 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:20.215190    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:20.488359 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:20.488335    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"]
Sep 09 03:40:20.490714 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:40:20.490683    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49773d57_b97f_40dc_b214_d2480b8990f1.slice/crio-976c67be63c4f5f03855168e5db4b055a9c73a71b15f06578e95668bbe912452 WatchSource:0}: Error finding container 976c67be63c4f5f03855168e5db4b055a9c73a71b15f06578e95668bbe912452: Status 404 returned error can't find the container with id 976c67be63c4f5f03855168e5db4b055a9c73a71b15f06578e95668bbe912452
Sep 09 03:40:21.064375 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:21.064344    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:21.371013 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:21.370980    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm" event={"ID":"49773d57-b97f-40dc-b214-d2480b8990f1","Type":"ContainerStarted","Data":"0c6f56f055d42ce69fb7c35895010422e7e326138ee494c49c5552da78cfdf0a"}
Sep 09 03:40:21.371013 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:21.371022    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm" event={"ID":"49773d57-b97f-40dc-b214-d2480b8990f1","Type":"ContainerStarted","Data":"976c67be63c4f5f03855168e5db4b055a9c73a71b15f06578e95668bbe912452"}
Sep 09 03:40:21.388847 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:21.388794    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm" podStartSLOduration=2.388774277 podStartE2EDuration="2.388774277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:40:21.388696235 +0000 UTC m=+1638.525444102" watchObservedRunningTime="2026-09-09 03:40:21.388774277 +0000 UTC m=+1638.525522142"
Sep 09 03:40:21.595874 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:21.595838    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:40:30.215542 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:30.215511    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:30.216001 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:30.215561    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:30.228257 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:30.228233    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:30.429637 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:30.429584    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:40:30.824747 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:30.824708    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:32.632295 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:32.632258    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:35.437374 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:35.437338    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:38.592196 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:38.592161    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:40:38.893206 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:38.893168    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:47.974339 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:47.974298    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:54.269902 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.269872    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"]
Sep 09 03:40:54.270309 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.270068    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g" podUID="3419d76e-0623-4f33-9c60-4ea7386fdbca" containerName="maas-api" containerID="cri-o://4272bf040b381f977bbbf057d2bc21d230eb3f1120cdb99e1ed13d33c72efb0b" gracePeriod=30
Sep 09 03:40:54.296387 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.296360    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:54.363259 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.363220    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:54.429070 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.429025    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:54.505219 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.505190    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:54.507511 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.507493    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:40:54.542861 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.542771    2560 generic.go:356] "Generic (PLEG): container finished" podID="3419d76e-0623-4f33-9c60-4ea7386fdbca" containerID="4272bf040b381f977bbbf057d2bc21d230eb3f1120cdb99e1ed13d33c72efb0b" exitCode=0
Sep 09 03:40:54.542861 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.542829    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g" event={"ID":"3419d76e-0623-4f33-9c60-4ea7386fdbca","Type":"ContainerDied","Data":"4272bf040b381f977bbbf057d2bc21d230eb3f1120cdb99e1ed13d33c72efb0b"}
Sep 09 03:40:54.542861 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.542839    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"
Sep 09 03:40:54.543120 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.542869    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g" event={"ID":"3419d76e-0623-4f33-9c60-4ea7386fdbca","Type":"ContainerDied","Data":"9d7e408515fed650c0f31f15d7724c44a4415c808cdf23c78776d1b974afc774"}
Sep 09 03:40:54.543120 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.542892    2560 scope.go:117] "RemoveContainer" containerID="4272bf040b381f977bbbf057d2bc21d230eb3f1120cdb99e1ed13d33c72efb0b"
Sep 09 03:40:54.555019 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.554995    2560 scope.go:117] "RemoveContainer" containerID="4272bf040b381f977bbbf057d2bc21d230eb3f1120cdb99e1ed13d33c72efb0b"
Sep 09 03:40:54.555356 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:40:54.555334    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4272bf040b381f977bbbf057d2bc21d230eb3f1120cdb99e1ed13d33c72efb0b\": container with ID starting with 4272bf040b381f977bbbf057d2bc21d230eb3f1120cdb99e1ed13d33c72efb0b not found: ID does not exist" containerID="4272bf040b381f977bbbf057d2bc21d230eb3f1120cdb99e1ed13d33c72efb0b"
Sep 09 03:40:54.555406 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.555368    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4272bf040b381f977bbbf057d2bc21d230eb3f1120cdb99e1ed13d33c72efb0b"} err="failed to get container status \"4272bf040b381f977bbbf057d2bc21d230eb3f1120cdb99e1ed13d33c72efb0b\": rpc error: code = NotFound desc = could not find container \"4272bf040b381f977bbbf057d2bc21d230eb3f1120cdb99e1ed13d33c72efb0b\": container with ID starting with 4272bf040b381f977bbbf057d2bc21d230eb3f1120cdb99e1ed13d33c72efb0b not found: ID does not exist"
Sep 09 03:40:54.611423 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.611391    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3419d76e-0623-4f33-9c60-4ea7386fdbca-openshift-service-ca\") pod \"3419d76e-0623-4f33-9c60-4ea7386fdbca\" (UID: \"3419d76e-0623-4f33-9c60-4ea7386fdbca\") "
Sep 09 03:40:54.611653 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.611449    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmrzv\" (UniqueName: \"kubernetes.io/projected/3419d76e-0623-4f33-9c60-4ea7386fdbca-kube-api-access-lmrzv\") pod \"3419d76e-0623-4f33-9c60-4ea7386fdbca\" (UID: \"3419d76e-0623-4f33-9c60-4ea7386fdbca\") "
Sep 09 03:40:54.611653 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.611493    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3419d76e-0623-4f33-9c60-4ea7386fdbca-maas-api-tls\") pod \"3419d76e-0623-4f33-9c60-4ea7386fdbca\" (UID: \"3419d76e-0623-4f33-9c60-4ea7386fdbca\") "
Sep 09 03:40:54.611653 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.611552    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3419d76e-0623-4f33-9c60-4ea7386fdbca-metrics-certs\") pod \"3419d76e-0623-4f33-9c60-4ea7386fdbca\" (UID: \"3419d76e-0623-4f33-9c60-4ea7386fdbca\") "
Sep 09 03:40:54.611837 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.611769    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3419d76e-0623-4f33-9c60-4ea7386fdbca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3419d76e-0623-4f33-9c60-4ea7386fdbca" (UID: "3419d76e-0623-4f33-9c60-4ea7386fdbca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:40:54.611956 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.611941    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3419d76e-0623-4f33-9c60-4ea7386fdbca-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:40:54.613710 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.613682    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3419d76e-0623-4f33-9c60-4ea7386fdbca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3419d76e-0623-4f33-9c60-4ea7386fdbca" (UID: "3419d76e-0623-4f33-9c60-4ea7386fdbca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:40:54.613710 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.613696    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3419d76e-0623-4f33-9c60-4ea7386fdbca-kube-api-access-lmrzv" (OuterVolumeSpecName: "kube-api-access-lmrzv") pod "3419d76e-0623-4f33-9c60-4ea7386fdbca" (UID: "3419d76e-0623-4f33-9c60-4ea7386fdbca"). InnerVolumeSpecName "kube-api-access-lmrzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:40:54.613858 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.613737    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3419d76e-0623-4f33-9c60-4ea7386fdbca-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3419d76e-0623-4f33-9c60-4ea7386fdbca" (UID: "3419d76e-0623-4f33-9c60-4ea7386fdbca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:40:54.713303 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.713267    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3419d76e-0623-4f33-9c60-4ea7386fdbca-metrics-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:40:54.713303 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.713298    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmrzv\" (UniqueName: \"kubernetes.io/projected/3419d76e-0623-4f33-9c60-4ea7386fdbca-kube-api-access-lmrzv\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:40:54.713303 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.713307    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3419d76e-0623-4f33-9c60-4ea7386fdbca-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:40:54.867504 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.867473    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"]
Sep 09 03:40:54.871658 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.871634    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f0891-845c88967d-sfd8g"]
Sep 09 03:40:54.945126 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:54.945098    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:55.359435 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:55.359399    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3419d76e-0623-4f33-9c60-4ea7386fdbca" path="/var/lib/kubelet/pods/3419d76e-0623-4f33-9c60-4ea7386fdbca/volumes"
Sep 09 03:40:58.741061 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:58.740996    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:40:59.106575 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:40:59.106541    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:41:00.850527 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:00.850493    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:00.953957 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:00.953922    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:01.244543 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:01.244502    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:01.775438 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:01.775404    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:15.442692 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:15.442657    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:17.269301 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.269264    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"]
Sep 09 03:41:17.269712 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.269509    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98" podUID="cbe82c0b-8207-4344-96db-20d64785238d" containerName="maas-api" containerID="cri-o://4e4e38047f64cc022053185b4012baa709208bcbd6ae974c7f3336719ec8a1e3" gracePeriod=30
Sep 09 03:41:17.305692 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.305661    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:17.384424 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.384396    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:17.489174 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.489136    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:17.518101 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.518072    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:41:17.547286 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.547209    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dghcq\" (UniqueName: \"kubernetes.io/projected/cbe82c0b-8207-4344-96db-20d64785238d-kube-api-access-dghcq\") pod \"cbe82c0b-8207-4344-96db-20d64785238d\" (UID: \"cbe82c0b-8207-4344-96db-20d64785238d\") "
Sep 09 03:41:17.547286 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.547277    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbe82c0b-8207-4344-96db-20d64785238d-maas-api-tls\") pod \"cbe82c0b-8207-4344-96db-20d64785238d\" (UID: \"cbe82c0b-8207-4344-96db-20d64785238d\") "
Sep 09 03:41:17.547482 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.547316    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe82c0b-8207-4344-96db-20d64785238d-openshift-service-ca\") pod \"cbe82c0b-8207-4344-96db-20d64785238d\" (UID: \"cbe82c0b-8207-4344-96db-20d64785238d\") "
Sep 09 03:41:17.547482 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.547363    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe82c0b-8207-4344-96db-20d64785238d-metrics-certs\") pod \"cbe82c0b-8207-4344-96db-20d64785238d\" (UID: \"cbe82c0b-8207-4344-96db-20d64785238d\") "
Sep 09 03:41:17.547808 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.547780    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe82c0b-8207-4344-96db-20d64785238d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cbe82c0b-8207-4344-96db-20d64785238d" (UID: "cbe82c0b-8207-4344-96db-20d64785238d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:41:17.549707 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.549682    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbe82c0b-8207-4344-96db-20d64785238d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cbe82c0b-8207-4344-96db-20d64785238d" (UID: "cbe82c0b-8207-4344-96db-20d64785238d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:41:17.550778 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.550747    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbe82c0b-8207-4344-96db-20d64785238d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cbe82c0b-8207-4344-96db-20d64785238d" (UID: "cbe82c0b-8207-4344-96db-20d64785238d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:41:17.550888 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.550856    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe82c0b-8207-4344-96db-20d64785238d-kube-api-access-dghcq" (OuterVolumeSpecName: "kube-api-access-dghcq") pod "cbe82c0b-8207-4344-96db-20d64785238d" (UID: "cbe82c0b-8207-4344-96db-20d64785238d"). InnerVolumeSpecName "kube-api-access-dghcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:41:17.585844 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.585810    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:17.648611 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.648559    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dghcq\" (UniqueName: \"kubernetes.io/projected/cbe82c0b-8207-4344-96db-20d64785238d-kube-api-access-dghcq\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:41:17.648611 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.648587    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbe82c0b-8207-4344-96db-20d64785238d-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:41:17.648611 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.648612    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe82c0b-8207-4344-96db-20d64785238d-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:41:17.648611 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.648620    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe82c0b-8207-4344-96db-20d64785238d-metrics-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:41:17.664308 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.664271    2560 generic.go:356] "Generic (PLEG): container finished" podID="cbe82c0b-8207-4344-96db-20d64785238d" containerID="4e4e38047f64cc022053185b4012baa709208bcbd6ae974c7f3336719ec8a1e3" exitCode=0
Sep 09 03:41:17.664489 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.664323    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"
Sep 09 03:41:17.664489 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.664337    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98" event={"ID":"cbe82c0b-8207-4344-96db-20d64785238d","Type":"ContainerDied","Data":"4e4e38047f64cc022053185b4012baa709208bcbd6ae974c7f3336719ec8a1e3"}
Sep 09 03:41:17.664489 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.664390    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98" event={"ID":"cbe82c0b-8207-4344-96db-20d64785238d","Type":"ContainerDied","Data":"3262309c9cd98d02372bddd92c3bd8fd7dd8d64c25a49a645671c492ad66f70e"}
Sep 09 03:41:17.664489 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.664420    2560 scope.go:117] "RemoveContainer" containerID="4e4e38047f64cc022053185b4012baa709208bcbd6ae974c7f3336719ec8a1e3"
Sep 09 03:41:17.674328 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.674308    2560 scope.go:117] "RemoveContainer" containerID="4e4e38047f64cc022053185b4012baa709208bcbd6ae974c7f3336719ec8a1e3"
Sep 09 03:41:17.674610 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:41:17.674573    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4e38047f64cc022053185b4012baa709208bcbd6ae974c7f3336719ec8a1e3\": container with ID starting with 4e4e38047f64cc022053185b4012baa709208bcbd6ae974c7f3336719ec8a1e3 not found: ID does not exist" containerID="4e4e38047f64cc022053185b4012baa709208bcbd6ae974c7f3336719ec8a1e3"
Sep 09 03:41:17.674664 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.674618    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4e38047f64cc022053185b4012baa709208bcbd6ae974c7f3336719ec8a1e3"} err="failed to get container status \"4e4e38047f64cc022053185b4012baa709208bcbd6ae974c7f3336719ec8a1e3\": rpc error: code = NotFound desc = could not find container \"4e4e38047f64cc022053185b4012baa709208bcbd6ae974c7f3336719ec8a1e3\": container with ID starting with 4e4e38047f64cc022053185b4012baa709208bcbd6ae974c7f3336719ec8a1e3 not found: ID does not exist"
Sep 09 03:41:17.688547 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.688523    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"]
Sep 09 03:41:17.694934 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:17.694910    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c4cb1-5948bb4cf4-trg98"]
Sep 09 03:41:18.071826 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:18.071791    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:18.642575 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:18.642530    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:19.358957 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:19.358914    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbe82c0b-8207-4344-96db-20d64785238d" path="/var/lib/kubelet/pods/cbe82c0b-8207-4344-96db-20d64785238d/volumes"
Sep 09 03:41:21.500620 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:21.500578    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:41:23.861301 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:23.861264    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:23.930900 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:23.930866    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:24.155879 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:24.155785    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:24.724931 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:24.724857    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:34.973061 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:34.972981    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"]
Sep 09 03:41:34.973637 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:34.973241    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb" podUID="963cd7f1-7fed-459a-8dff-96f5b992e31b" containerName="maas-api" containerID="cri-o://c27c3d5bd54af095d3efa1193a09009664855c56e7412576bc74aaeff43fe0e8" gracePeriod=30
Sep 09 03:41:35.006276 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.006215    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:35.106237 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.106201    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:35.207559 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.207524    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:35.225569 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.225513    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:41:35.275755 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.275718    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:35.315290 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.315257    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963cd7f1-7fed-459a-8dff-96f5b992e31b-openshift-service-ca\") pod \"963cd7f1-7fed-459a-8dff-96f5b992e31b\" (UID: \"963cd7f1-7fed-459a-8dff-96f5b992e31b\") "
Sep 09 03:41:35.315465 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.315343    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsghl\" (UniqueName: \"kubernetes.io/projected/963cd7f1-7fed-459a-8dff-96f5b992e31b-kube-api-access-zsghl\") pod \"963cd7f1-7fed-459a-8dff-96f5b992e31b\" (UID: \"963cd7f1-7fed-459a-8dff-96f5b992e31b\") "
Sep 09 03:41:35.315465 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.315375    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963cd7f1-7fed-459a-8dff-96f5b992e31b-maas-api-tls\") pod \"963cd7f1-7fed-459a-8dff-96f5b992e31b\" (UID: \"963cd7f1-7fed-459a-8dff-96f5b992e31b\") "
Sep 09 03:41:35.315465 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.315422    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963cd7f1-7fed-459a-8dff-96f5b992e31b-metrics-certs\") pod \"963cd7f1-7fed-459a-8dff-96f5b992e31b\" (UID: \"963cd7f1-7fed-459a-8dff-96f5b992e31b\") "
Sep 09 03:41:35.315682 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.315657    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963cd7f1-7fed-459a-8dff-96f5b992e31b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "963cd7f1-7fed-459a-8dff-96f5b992e31b" (UID: "963cd7f1-7fed-459a-8dff-96f5b992e31b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:41:35.315817 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.315800    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963cd7f1-7fed-459a-8dff-96f5b992e31b-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:41:35.317501 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.317470    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963cd7f1-7fed-459a-8dff-96f5b992e31b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "963cd7f1-7fed-459a-8dff-96f5b992e31b" (UID: "963cd7f1-7fed-459a-8dff-96f5b992e31b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:41:35.317623 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.317532    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963cd7f1-7fed-459a-8dff-96f5b992e31b-kube-api-access-zsghl" (OuterVolumeSpecName: "kube-api-access-zsghl") pod "963cd7f1-7fed-459a-8dff-96f5b992e31b" (UID: "963cd7f1-7fed-459a-8dff-96f5b992e31b"). InnerVolumeSpecName "kube-api-access-zsghl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:41:35.317623 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.317554    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963cd7f1-7fed-459a-8dff-96f5b992e31b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "963cd7f1-7fed-459a-8dff-96f5b992e31b" (UID: "963cd7f1-7fed-459a-8dff-96f5b992e31b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:41:35.417065 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.417035    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963cd7f1-7fed-459a-8dff-96f5b992e31b-metrics-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:41:35.417065 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.417063    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsghl\" (UniqueName: \"kubernetes.io/projected/963cd7f1-7fed-459a-8dff-96f5b992e31b-kube-api-access-zsghl\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:41:35.417253 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.417074    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963cd7f1-7fed-459a-8dff-96f5b992e31b-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:41:35.607247 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.607210    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:35.764877 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.764821    2560 generic.go:356] "Generic (PLEG): container finished" podID="963cd7f1-7fed-459a-8dff-96f5b992e31b" containerID="c27c3d5bd54af095d3efa1193a09009664855c56e7412576bc74aaeff43fe0e8" exitCode=0
Sep 09 03:41:35.764877 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.764878    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"
Sep 09 03:41:35.764877 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.764878    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb" event={"ID":"963cd7f1-7fed-459a-8dff-96f5b992e31b","Type":"ContainerDied","Data":"c27c3d5bd54af095d3efa1193a09009664855c56e7412576bc74aaeff43fe0e8"}
Sep 09 03:41:35.765218 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.764932    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb" event={"ID":"963cd7f1-7fed-459a-8dff-96f5b992e31b","Type":"ContainerDied","Data":"9039aeed27604f8b130671c54e28a640d4ab017655f4e10d691b97d832f0f70c"}
Sep 09 03:41:35.765218 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.764961    2560 scope.go:117] "RemoveContainer" containerID="c27c3d5bd54af095d3efa1193a09009664855c56e7412576bc74aaeff43fe0e8"
Sep 09 03:41:35.775693 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.775673    2560 scope.go:117] "RemoveContainer" containerID="c27c3d5bd54af095d3efa1193a09009664855c56e7412576bc74aaeff43fe0e8"
Sep 09 03:41:35.775960 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:41:35.775940    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c27c3d5bd54af095d3efa1193a09009664855c56e7412576bc74aaeff43fe0e8\": container with ID starting with c27c3d5bd54af095d3efa1193a09009664855c56e7412576bc74aaeff43fe0e8 not found: ID does not exist" containerID="c27c3d5bd54af095d3efa1193a09009664855c56e7412576bc74aaeff43fe0e8"
Sep 09 03:41:35.776036 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.775965    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27c3d5bd54af095d3efa1193a09009664855c56e7412576bc74aaeff43fe0e8"} err="failed to get container status \"c27c3d5bd54af095d3efa1193a09009664855c56e7412576bc74aaeff43fe0e8\": rpc error: code = NotFound desc = could not find container \"c27c3d5bd54af095d3efa1193a09009664855c56e7412576bc74aaeff43fe0e8\": container with ID starting with c27c3d5bd54af095d3efa1193a09009664855c56e7412576bc74aaeff43fe0e8 not found: ID does not exist"
Sep 09 03:41:35.791817 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.791787    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"]
Sep 09 03:41:35.799269 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:35.799247    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-eca017-7f9bbc6cfc-5cqrb"]
Sep 09 03:41:37.359044 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:37.359010    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="963cd7f1-7fed-459a-8dff-96f5b992e31b" path="/var/lib/kubelet/pods/963cd7f1-7fed-459a-8dff-96f5b992e31b/volumes"
Sep 09 03:41:40.390200 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:40.390166    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:41:41.588429 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:41.588398    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:41.672382 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:41.672349    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:41.899971 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:41.899939    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:42.438323 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:42.438290    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:52.841983 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:52.841942    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"]
Sep 09 03:41:52.844055 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:52.842189    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx" podUID="5413b949-e629-4b13-878c-6ea8480c1cf8" containerName="maas-api" containerID="cri-o://306aa7bd73340688956ad250afafeabd494f99341b2070770b5f93c4e983ada6" gracePeriod=30
Sep 09 03:41:52.870031 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:52.869998    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:52.891929 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:52.891893    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb"]
Sep 09 03:41:52.892481 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:52.892440    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb" podUID="2f1e0d6e-d9bb-4125-a8a6-f26da5242731" containerName="payload-processing" containerID="cri-o://3c211a05d7db90f2fd96b208fdfceaeb8c8bf170aa7407e5c8d2b46a1ac4cadb" gracePeriod=30
Sep 09 03:41:52.941019 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:52.940986    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:53.039744 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.039689    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:53.110906 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.110871    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:53.159669 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.159646    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:41:53.162233 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.162212    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb"
Sep 09 03:41:53.178207 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.178190    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1e0d6e-d9bb-4125-a8a6-f26da5242731-plugins-config-volume\") pod \"2f1e0d6e-d9bb-4125-a8a6-f26da5242731\" (UID: \"2f1e0d6e-d9bb-4125-a8a6-f26da5242731\") "
Sep 09 03:41:53.178288 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.178236    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5413b949-e629-4b13-878c-6ea8480c1cf8-openshift-service-ca\") pod \"5413b949-e629-4b13-878c-6ea8480c1cf8\" (UID: \"5413b949-e629-4b13-878c-6ea8480c1cf8\") "
Sep 09 03:41:53.178288 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.178263    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5413b949-e629-4b13-878c-6ea8480c1cf8-maas-api-tls\") pod \"5413b949-e629-4b13-878c-6ea8480c1cf8\" (UID: \"5413b949-e629-4b13-878c-6ea8480c1cf8\") "
Sep 09 03:41:53.178410 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.178289    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmspx\" (UniqueName: \"kubernetes.io/projected/2f1e0d6e-d9bb-4125-a8a6-f26da5242731-kube-api-access-hmspx\") pod \"2f1e0d6e-d9bb-4125-a8a6-f26da5242731\" (UID: \"2f1e0d6e-d9bb-4125-a8a6-f26da5242731\") "
Sep 09 03:41:53.178410 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.178312    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5413b949-e629-4b13-878c-6ea8480c1cf8-metrics-certs\") pod \"5413b949-e629-4b13-878c-6ea8480c1cf8\" (UID: \"5413b949-e629-4b13-878c-6ea8480c1cf8\") "
Sep 09 03:41:53.178522 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.178408    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4djf\" (UniqueName: \"kubernetes.io/projected/5413b949-e629-4b13-878c-6ea8480c1cf8-kube-api-access-q4djf\") pod \"5413b949-e629-4b13-878c-6ea8480c1cf8\" (UID: \"5413b949-e629-4b13-878c-6ea8480c1cf8\") "
Sep 09 03:41:53.178611 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.178569    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5413b949-e629-4b13-878c-6ea8480c1cf8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5413b949-e629-4b13-878c-6ea8480c1cf8" (UID: "5413b949-e629-4b13-878c-6ea8480c1cf8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:41:53.178680 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.178656    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1e0d6e-d9bb-4125-a8a6-f26da5242731-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2f1e0d6e-d9bb-4125-a8a6-f26da5242731" (UID: "2f1e0d6e-d9bb-4125-a8a6-f26da5242731"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:41:53.178790 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.178775    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1e0d6e-d9bb-4125-a8a6-f26da5242731-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:41:53.178838 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.178796    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5413b949-e629-4b13-878c-6ea8480c1cf8-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:41:53.180743 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.180717    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5413b949-e629-4b13-878c-6ea8480c1cf8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5413b949-e629-4b13-878c-6ea8480c1cf8" (UID: "5413b949-e629-4b13-878c-6ea8480c1cf8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:41:53.180743 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.180725    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1e0d6e-d9bb-4125-a8a6-f26da5242731-kube-api-access-hmspx" (OuterVolumeSpecName: "kube-api-access-hmspx") pod "2f1e0d6e-d9bb-4125-a8a6-f26da5242731" (UID: "2f1e0d6e-d9bb-4125-a8a6-f26da5242731"). InnerVolumeSpecName "kube-api-access-hmspx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:41:53.180976 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.180737    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5413b949-e629-4b13-878c-6ea8480c1cf8-kube-api-access-q4djf" (OuterVolumeSpecName: "kube-api-access-q4djf") pod "5413b949-e629-4b13-878c-6ea8480c1cf8" (UID: "5413b949-e629-4b13-878c-6ea8480c1cf8"). InnerVolumeSpecName "kube-api-access-q4djf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:41:53.180976 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.180948    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5413b949-e629-4b13-878c-6ea8480c1cf8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5413b949-e629-4b13-878c-6ea8480c1cf8" (UID: "5413b949-e629-4b13-878c-6ea8480c1cf8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:41:53.279814 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.279777    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5413b949-e629-4b13-878c-6ea8480c1cf8-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:41:53.279814 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.279810    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmspx\" (UniqueName: \"kubernetes.io/projected/2f1e0d6e-d9bb-4125-a8a6-f26da5242731-kube-api-access-hmspx\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:41:53.279814 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.279820    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5413b949-e629-4b13-878c-6ea8480c1cf8-metrics-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:41:53.280043 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.279828    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4djf\" (UniqueName: \"kubernetes.io/projected/5413b949-e629-4b13-878c-6ea8480c1cf8-kube-api-access-q4djf\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:41:53.550927 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.550866    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:53.867726 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.867695    2560 generic.go:356] "Generic (PLEG): container finished" podID="2f1e0d6e-d9bb-4125-a8a6-f26da5242731" containerID="3c211a05d7db90f2fd96b208fdfceaeb8c8bf170aa7407e5c8d2b46a1ac4cadb" exitCode=0
Sep 09 03:41:53.868115 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.867742    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb"
Sep 09 03:41:53.868115 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.867783    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb" event={"ID":"2f1e0d6e-d9bb-4125-a8a6-f26da5242731","Type":"ContainerDied","Data":"3c211a05d7db90f2fd96b208fdfceaeb8c8bf170aa7407e5c8d2b46a1ac4cadb"}
Sep 09 03:41:53.868115 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.867849    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb" event={"ID":"2f1e0d6e-d9bb-4125-a8a6-f26da5242731","Type":"ContainerDied","Data":"3fed1a00814a72d3c7152367651941ce7d1854c753a0c86eb993795d405a05a0"}
Sep 09 03:41:53.868115 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.867882    2560 scope.go:117] "RemoveContainer" containerID="3c211a05d7db90f2fd96b208fdfceaeb8c8bf170aa7407e5c8d2b46a1ac4cadb"
Sep 09 03:41:53.869311 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.869288    2560 generic.go:356] "Generic (PLEG): container finished" podID="5413b949-e629-4b13-878c-6ea8480c1cf8" containerID="306aa7bd73340688956ad250afafeabd494f99341b2070770b5f93c4e983ada6" exitCode=0
Sep 09 03:41:53.869369 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.869337    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"
Sep 09 03:41:53.869411 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.869372    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx" event={"ID":"5413b949-e629-4b13-878c-6ea8480c1cf8","Type":"ContainerDied","Data":"306aa7bd73340688956ad250afafeabd494f99341b2070770b5f93c4e983ada6"}
Sep 09 03:41:53.869451 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.869413    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx" event={"ID":"5413b949-e629-4b13-878c-6ea8480c1cf8","Type":"ContainerDied","Data":"9233ab9e7df33dcbd121b4fc65bcfc3cf6d4d3d86aab965625e19c5e726e01d6"}
Sep 09 03:41:53.876660 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.876645    2560 scope.go:117] "RemoveContainer" containerID="3c211a05d7db90f2fd96b208fdfceaeb8c8bf170aa7407e5c8d2b46a1ac4cadb"
Sep 09 03:41:53.876940 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:41:53.876922    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c211a05d7db90f2fd96b208fdfceaeb8c8bf170aa7407e5c8d2b46a1ac4cadb\": container with ID starting with 3c211a05d7db90f2fd96b208fdfceaeb8c8bf170aa7407e5c8d2b46a1ac4cadb not found: ID does not exist" containerID="3c211a05d7db90f2fd96b208fdfceaeb8c8bf170aa7407e5c8d2b46a1ac4cadb"
Sep 09 03:41:53.877012 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.876947    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c211a05d7db90f2fd96b208fdfceaeb8c8bf170aa7407e5c8d2b46a1ac4cadb"} err="failed to get container status \"3c211a05d7db90f2fd96b208fdfceaeb8c8bf170aa7407e5c8d2b46a1ac4cadb\": rpc error: code = NotFound desc = could not find container \"3c211a05d7db90f2fd96b208fdfceaeb8c8bf170aa7407e5c8d2b46a1ac4cadb\": container with ID starting with 3c211a05d7db90f2fd96b208fdfceaeb8c8bf170aa7407e5c8d2b46a1ac4cadb not found: ID does not exist"
Sep 09 03:41:53.877012 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.876964    2560 scope.go:117] "RemoveContainer" containerID="306aa7bd73340688956ad250afafeabd494f99341b2070770b5f93c4e983ada6"
Sep 09 03:41:53.885276 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.885259    2560 scope.go:117] "RemoveContainer" containerID="306aa7bd73340688956ad250afafeabd494f99341b2070770b5f93c4e983ada6"
Sep 09 03:41:53.885565 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:41:53.885542    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306aa7bd73340688956ad250afafeabd494f99341b2070770b5f93c4e983ada6\": container with ID starting with 306aa7bd73340688956ad250afafeabd494f99341b2070770b5f93c4e983ada6 not found: ID does not exist" containerID="306aa7bd73340688956ad250afafeabd494f99341b2070770b5f93c4e983ada6"
Sep 09 03:41:53.885674 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.885571    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306aa7bd73340688956ad250afafeabd494f99341b2070770b5f93c4e983ada6"} err="failed to get container status \"306aa7bd73340688956ad250afafeabd494f99341b2070770b5f93c4e983ada6\": rpc error: code = NotFound desc = could not find container \"306aa7bd73340688956ad250afafeabd494f99341b2070770b5f93c4e983ada6\": container with ID starting with 306aa7bd73340688956ad250afafeabd494f99341b2070770b5f93c4e983ada6 not found: ID does not exist"
Sep 09 03:41:53.887658 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.887637    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"]
Sep 09 03:41:53.889410 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.889392    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0d0cae-867d87d8d4-7h6tx"]
Sep 09 03:41:53.900649 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.900629    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb"]
Sep 09 03:41:53.905220 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:53.905198    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0d0cae-85f669d5c-2bxlb"]
Sep 09 03:41:54.499034 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:54.498991    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:41:55.358583 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:55.358546    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f1e0d6e-d9bb-4125-a8a6-f26da5242731" path="/var/lib/kubelet/pods/2f1e0d6e-d9bb-4125-a8a6-f26da5242731/volumes"
Sep 09 03:41:55.358974 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:55.358960    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5413b949-e629-4b13-878c-6ea8480c1cf8" path="/var/lib/kubelet/pods/5413b949-e629-4b13-878c-6ea8480c1cf8/volumes"
Sep 09 03:41:59.349059 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:59.349025    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:59.419937 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:59.419907    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:41:59.658513 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:41:59.658479    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:04.388085 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:04.388046    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:42:10.714361 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:10.714315    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"]
Sep 09 03:42:10.737889 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:10.737823    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:11.813565 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:11.813528    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:12.280238 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:12.280206    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:12.591141 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:12.591056    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:12.896288 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:12.896260    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:15.505282 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:15.505241    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:42:16.248207 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:16.248174    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:16.327915 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:16.327879    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:17.418173 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.418138    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"]
Sep 09 03:42:17.418561 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.418539    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5413b949-e629-4b13-878c-6ea8480c1cf8" containerName="maas-api"
Sep 09 03:42:17.418561 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.418552    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5413b949-e629-4b13-878c-6ea8480c1cf8" containerName="maas-api"
Sep 09 03:42:17.418561 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.418562    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="963cd7f1-7fed-459a-8dff-96f5b992e31b" containerName="maas-api"
Sep 09 03:42:17.418718 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.418568    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="963cd7f1-7fed-459a-8dff-96f5b992e31b" containerName="maas-api"
Sep 09 03:42:17.418718 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.418575    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1e0d6e-d9bb-4125-a8a6-f26da5242731" containerName="payload-processing"
Sep 09 03:42:17.418718 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.418581    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1e0d6e-d9bb-4125-a8a6-f26da5242731" containerName="payload-processing"
Sep 09 03:42:17.418718 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.418607    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbe82c0b-8207-4344-96db-20d64785238d" containerName="maas-api"
Sep 09 03:42:17.418718 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.418612    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbe82c0b-8207-4344-96db-20d64785238d" containerName="maas-api"
Sep 09 03:42:17.418718 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.418660    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3419d76e-0623-4f33-9c60-4ea7386fdbca" containerName="maas-api"
Sep 09 03:42:17.418718 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.418668    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3419d76e-0623-4f33-9c60-4ea7386fdbca" containerName="maas-api"
Sep 09 03:42:17.418952 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.418759    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="963cd7f1-7fed-459a-8dff-96f5b992e31b" containerName="maas-api"
Sep 09 03:42:17.418952 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.418770    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3419d76e-0623-4f33-9c60-4ea7386fdbca" containerName="maas-api"
Sep 09 03:42:17.418952 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.418784    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5413b949-e629-4b13-878c-6ea8480c1cf8" containerName="maas-api"
Sep 09 03:42:17.418952 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.418794    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbe82c0b-8207-4344-96db-20d64785238d" containerName="maas-api"
Sep 09 03:42:17.418952 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.418804    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f1e0d6e-d9bb-4125-a8a6-f26da5242731" containerName="payload-processing"
Sep 09 03:42:17.421984 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.421962    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:42:17.426041 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.426021    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-46be70-serving-cert\""
Sep 09 03:42:17.447115 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.447096    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"]
Sep 09 03:42:17.516439 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.516404    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq8vq\" (UniqueName: \"kubernetes.io/projected/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-kube-api-access-xq8vq\") pod \"maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w\" (UID: \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:42:17.516650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.516474    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w\" (UID: \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:42:17.516650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.516560    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-metrics-certs\") pod \"maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w\" (UID: \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:42:17.516650 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.516646    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w\" (UID: \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:42:17.563764 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.563729    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:17.618125 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.618096    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w\" (UID: \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:42:17.618322 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.618159    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq8vq\" (UniqueName: \"kubernetes.io/projected/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-kube-api-access-xq8vq\") pod \"maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w\" (UID: \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:42:17.618322 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.618236    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w\" (UID: \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:42:17.618322 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:42:17.618243    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-serving-cert: secret "maas-api-e2e-ipp-a-46be70-serving-cert" not found
Sep 09 03:42:17.618322 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:42:17.618300    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-maas-api-tls podName:b5bd6fff-fb27-4a76-a230-9d13b4cb74a2 nodeName:}" failed. No retries permitted until 2026-09-09 03:42:18.118284883 +0000 UTC m=+1755.255032730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-maas-api-tls") pod "maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w" (UID: "b5bd6fff-fb27-4a76-a230-9d13b4cb74a2") : secret "maas-api-e2e-ipp-a-46be70-serving-cert" not found
Sep 09 03:42:17.618322 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.618321    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-metrics-certs\") pod \"maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w\" (UID: \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:42:17.618975 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.618952    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w\" (UID: \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:42:17.620718 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.620696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-metrics-certs\") pod \"maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w\" (UID: \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:42:17.628452 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.628424    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq8vq\" (UniqueName: \"kubernetes.io/projected/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-kube-api-access-xq8vq\") pod \"maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w\" (UID: \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:42:17.661220 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.661193    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:17.976287 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:17.976250    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:18.082220 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:18.082192    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:18.122882 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:18.122856    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w\" (UID: \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:42:18.125248 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:18.125224    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w\" (UID: \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:42:18.184610 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:18.184558    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:18.284672 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:18.284580    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:18.332358 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:18.332327    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:42:18.606546 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:18.606516    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"]
Sep 09 03:42:18.608691 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:42:18.608653    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bd6fff_fb27_4a76_a230_9d13b4cb74a2.slice/crio-d09f49d446ab8b705f2fd8ab0e59af1165081c4fe2d02c296919c0edae20bee9 WatchSource:0}: Error finding container d09f49d446ab8b705f2fd8ab0e59af1165081c4fe2d02c296919c0edae20bee9: Status 404 returned error can't find the container with id d09f49d446ab8b705f2fd8ab0e59af1165081c4fe2d02c296919c0edae20bee9
Sep 09 03:42:18.999838 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:18.999754    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w" event={"ID":"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2","Type":"ContainerStarted","Data":"d09f49d446ab8b705f2fd8ab0e59af1165081c4fe2d02c296919c0edae20bee9"}
Sep 09 03:42:20.006721 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:20.006683    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w" event={"ID":"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2","Type":"ContainerStarted","Data":"0fddacfe486a73a9ad05892ba8638e6321a80995f9fadb8981dec46209e7dc37"}
Sep 09 03:42:20.007106 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:20.006781    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:42:20.031779 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:20.031721    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w" podStartSLOduration=2.48085217 podStartE2EDuration="3.031707007s" totalImagesPullingTime="550.854837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:42:20.029565349 +0000 UTC m=+1757.166313215" watchObservedRunningTime="2026-09-09 03:42:20.031707007 +0000 UTC m=+1757.168454872"
Sep 09 03:42:25.734388 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:25.734316    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm" podUID="49773d57-b97f-40dc-b214-d2480b8990f1" containerName="main" containerID="cri-o://0c6f56f055d42ce69fb7c35895010422e7e326138ee494c49c5552da78cfdf0a" gracePeriod=45
Sep 09 03:42:25.991443 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:25.991387    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:42:26.017857 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.017827    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:42:26.043575 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.043546    2560 generic.go:356] "Generic (PLEG): container finished" podID="49773d57-b97f-40dc-b214-d2480b8990f1" containerID="0c6f56f055d42ce69fb7c35895010422e7e326138ee494c49c5552da78cfdf0a" exitCode=0
Sep 09 03:42:26.043712 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.043582    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"
Sep 09 03:42:26.043712 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.043628    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm" event={"ID":"49773d57-b97f-40dc-b214-d2480b8990f1","Type":"ContainerDied","Data":"0c6f56f055d42ce69fb7c35895010422e7e326138ee494c49c5552da78cfdf0a"}
Sep 09 03:42:26.043712 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.043670    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm" event={"ID":"49773d57-b97f-40dc-b214-d2480b8990f1","Type":"ContainerDied","Data":"976c67be63c4f5f03855168e5db4b055a9c73a71b15f06578e95668bbe912452"}
Sep 09 03:42:26.043712 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.043690    2560 scope.go:117] "RemoveContainer" containerID="0c6f56f055d42ce69fb7c35895010422e7e326138ee494c49c5552da78cfdf0a"
Sep 09 03:42:26.053844 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.053829    2560 scope.go:117] "RemoveContainer" containerID="0c6f56f055d42ce69fb7c35895010422e7e326138ee494c49c5552da78cfdf0a"
Sep 09 03:42:26.054118 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:42:26.054096    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c6f56f055d42ce69fb7c35895010422e7e326138ee494c49c5552da78cfdf0a\": container with ID starting with 0c6f56f055d42ce69fb7c35895010422e7e326138ee494c49c5552da78cfdf0a not found: ID does not exist" containerID="0c6f56f055d42ce69fb7c35895010422e7e326138ee494c49c5552da78cfdf0a"
Sep 09 03:42:26.054197 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.054130    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6f56f055d42ce69fb7c35895010422e7e326138ee494c49c5552da78cfdf0a"} err="failed to get container status \"0c6f56f055d42ce69fb7c35895010422e7e326138ee494c49c5552da78cfdf0a\": rpc error: code = NotFound desc = could not find container \"0c6f56f055d42ce69fb7c35895010422e7e326138ee494c49c5552da78cfdf0a\": container with ID starting with 0c6f56f055d42ce69fb7c35895010422e7e326138ee494c49c5552da78cfdf0a not found: ID does not exist"
Sep 09 03:42:26.096435 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.096415    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-dshm\") pod \"49773d57-b97f-40dc-b214-d2480b8990f1\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") "
Sep 09 03:42:26.096560 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.096443    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-home\") pod \"49773d57-b97f-40dc-b214-d2480b8990f1\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") "
Sep 09 03:42:26.096560 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.096466    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-tmp-dir\") pod \"49773d57-b97f-40dc-b214-d2480b8990f1\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") "
Sep 09 03:42:26.096560 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.096522    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wj8k\" (UniqueName: \"kubernetes.io/projected/49773d57-b97f-40dc-b214-d2480b8990f1-kube-api-access-9wj8k\") pod \"49773d57-b97f-40dc-b214-d2480b8990f1\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") "
Sep 09 03:42:26.096746 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.096576    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-model-cache\") pod \"49773d57-b97f-40dc-b214-d2480b8990f1\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") "
Sep 09 03:42:26.096746 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.096617    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49773d57-b97f-40dc-b214-d2480b8990f1-tls-certs\") pod \"49773d57-b97f-40dc-b214-d2480b8990f1\" (UID: \"49773d57-b97f-40dc-b214-d2480b8990f1\") "
Sep 09 03:42:26.096746 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.096713    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-home" (OuterVolumeSpecName: "home") pod "49773d57-b97f-40dc-b214-d2480b8990f1" (UID: "49773d57-b97f-40dc-b214-d2480b8990f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:42:26.097263 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.097172    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "49773d57-b97f-40dc-b214-d2480b8990f1" (UID: "49773d57-b97f-40dc-b214-d2480b8990f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:42:26.098644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.097574    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-model-cache" (OuterVolumeSpecName: "model-cache") pod "49773d57-b97f-40dc-b214-d2480b8990f1" (UID: "49773d57-b97f-40dc-b214-d2480b8990f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:42:26.098644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.097875    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-model-cache\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:42:26.098644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.097912    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-home\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:42:26.098644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.097955    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-tmp-dir\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:42:26.101389 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.099647    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49773d57-b97f-40dc-b214-d2480b8990f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49773d57-b97f-40dc-b214-d2480b8990f1" (UID: "49773d57-b97f-40dc-b214-d2480b8990f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:42:26.105956 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.105781    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49773d57-b97f-40dc-b214-d2480b8990f1-kube-api-access-9wj8k" (OuterVolumeSpecName: "kube-api-access-9wj8k") pod "49773d57-b97f-40dc-b214-d2480b8990f1" (UID: "49773d57-b97f-40dc-b214-d2480b8990f1"). InnerVolumeSpecName "kube-api-access-9wj8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:42:26.106915 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.106893    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-dshm" (OuterVolumeSpecName: "dshm") pod "49773d57-b97f-40dc-b214-d2480b8990f1" (UID: "49773d57-b97f-40dc-b214-d2480b8990f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:42:26.199200 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.199177    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wj8k\" (UniqueName: \"kubernetes.io/projected/49773d57-b97f-40dc-b214-d2480b8990f1-kube-api-access-9wj8k\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:42:26.199200 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.199202    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49773d57-b97f-40dc-b214-d2480b8990f1-tls-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:42:26.199418 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.199214    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49773d57-b97f-40dc-b214-d2480b8990f1-dshm\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:42:26.369473 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.369445    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"]
Sep 09 03:42:26.373934 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:26.373908    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5d8e85/rate-test-model-5d8e85-kserve-799488d95-v7tlm"]
Sep 09 03:42:27.358828 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:27.358798    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49773d57-b97f-40dc-b214-d2480b8990f1" path="/var/lib/kubelet/pods/49773d57-b97f-40dc-b214-d2480b8990f1/volumes"
Sep 09 03:42:29.100878 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:29.100847    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:42:41.295062 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:41.295022    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:41.992568 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:41.992530    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:42.332856 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:42.332771    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:42.432937 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:42.432899    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:45.444552 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:45.444515    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:45.537277 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:45.537246    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:46.555303 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.555264    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"]
Sep 09 03:42:46.555737 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.555701    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49773d57-b97f-40dc-b214-d2480b8990f1" containerName="main"
Sep 09 03:42:46.555737 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.555713    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="49773d57-b97f-40dc-b214-d2480b8990f1" containerName="main"
Sep 09 03:42:46.555823 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.555781    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="49773d57-b97f-40dc-b214-d2480b8990f1" containerName="main"
Sep 09 03:42:46.559317 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.559288    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:42:46.562008 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.561985    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-60a731-serving-cert\""
Sep 09 03:42:46.569698 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.569675    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"]
Sep 09 03:42:46.623069 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.623027    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:46.681116 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.681075    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx\" (UID: \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:42:46.681116 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.681118    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx\" (UID: \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:42:46.681327 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.681151    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kznmd\" (UniqueName: \"kubernetes.io/projected/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-kube-api-access-kznmd\") pod \"maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx\" (UID: \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:42:46.681327 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.681166    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-metrics-certs\") pod \"maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx\" (UID: \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:42:46.720876 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.720837    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:46.782391 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.782339    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx\" (UID: \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:42:46.782391 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.782392    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx\" (UID: \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:42:46.782697 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.782438    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kznmd\" (UniqueName: \"kubernetes.io/projected/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-kube-api-access-kznmd\") pod \"maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx\" (UID: \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:42:46.782697 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.782463    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-metrics-certs\") pod \"maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx\" (UID: \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:42:46.782697 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:42:46.782541    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-serving-cert: secret "maas-api-e2e-ipp-b-60a731-serving-cert" not found
Sep 09 03:42:46.782697 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:42:46.782669    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-maas-api-tls podName:fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3 nodeName:}" failed. No retries permitted until 2026-09-09 03:42:47.282651372 +0000 UTC m=+1784.419399227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-maas-api-tls") pod "maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx" (UID: "fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3") : secret "maas-api-e2e-ipp-b-60a731-serving-cert" not found
Sep 09 03:42:46.783139 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.783114    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx\" (UID: \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:42:46.785091 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.785068    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-metrics-certs\") pod \"maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx\" (UID: \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:42:46.791907 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.791878    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kznmd\" (UniqueName: \"kubernetes.io/projected/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-kube-api-access-kznmd\") pod \"maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx\" (UID: \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:42:46.939898 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:46.939865    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:47.172557 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:47.172519    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:47.287267 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:47.287184    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx\" (UID: \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:42:47.289578 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:47.289544    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx\" (UID: \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:42:47.320291 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:47.320258    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:42:47.471811 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:47.471779    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:42:47.483800 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:47.483716    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c"]
Sep 09 03:42:47.487854 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:47.487832    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c"
Sep 09 03:42:47.490748 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:47.490726    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-60a731-dockercfg-wf2ht\""
Sep 09 03:42:47.490861 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:47.490840    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-60a731\""
Sep 09 03:42:47.506691 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:47.506420    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c"]
Sep 09 03:42:47.589498 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:47.589419    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9g9c\" (UniqueName: \"kubernetes.io/projected/d0aff821-b750-4f32-b217-a1b0173b5a73-kube-api-access-c9g9c\") pod \"payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c\" (UID: \"d0aff821-b750-4f32-b217-a1b0173b5a73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c"
Sep 09 03:42:47.589895 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:47.589585    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0aff821-b750-4f32-b217-a1b0173b5a73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c\" (UID: \"d0aff821-b750-4f32-b217-a1b0173b5a73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c"
Sep 09 03:42:47.690302 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:47.690263    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9g9c\" (UniqueName: \"kubernetes.io/projected/d0aff821-b750-4f32-b217-a1b0173b5a73-kube-api-access-c9g9c\") pod \"payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c\" (UID: \"d0aff821-b750-4f32-b217-a1b0173b5a73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c"
Sep 09 03:42:47.690473 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:47.690379    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0aff821-b750-4f32-b217-a1b0173b5a73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c\" (UID: \"d0aff821-b750-4f32-b217-a1b0173b5a73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c"
Sep 09 03:42:47.691057 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:47.691039    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0aff821-b750-4f32-b217-a1b0173b5a73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c\" (UID: \"d0aff821-b750-4f32-b217-a1b0173b5a73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c"
Sep 09 03:42:47.699450 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:47.699423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9g9c\" (UniqueName: \"kubernetes.io/projected/d0aff821-b750-4f32-b217-a1b0173b5a73-kube-api-access-c9g9c\") pod \"payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c\" (UID: \"d0aff821-b750-4f32-b217-a1b0173b5a73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c"
Sep 09 03:42:47.753056 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:47.753030    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"]
Sep 09 03:42:47.755563 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:42:47.755536    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0ca5ad_fc83_4bb8_b821_2650fb0dfff3.slice/crio-2e343c6a4a0bed2ef1566731706a0d2c8468ece84294b9d44fc8aa79d68d69a4 WatchSource:0}: Error finding container 2e343c6a4a0bed2ef1566731706a0d2c8468ece84294b9d44fc8aa79d68d69a4: Status 404 returned error can't find the container with id 2e343c6a4a0bed2ef1566731706a0d2c8468ece84294b9d44fc8aa79d68d69a4
Sep 09 03:42:47.811757 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:47.811726    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c"
Sep 09 03:42:48.087030 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:48.087003    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c"]
Sep 09 03:42:48.089377 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:42:48.089354    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0aff821_b750_4f32_b217_a1b0173b5a73.slice/crio-1cf7adf4ef829bcc3e85beef98826fce11db8879784b9db839bdfea0422d8067 WatchSource:0}: Error finding container 1cf7adf4ef829bcc3e85beef98826fce11db8879784b9db839bdfea0422d8067: Status 404 returned error can't find the container with id 1cf7adf4ef829bcc3e85beef98826fce11db8879784b9db839bdfea0422d8067
Sep 09 03:42:48.161504 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:48.161467    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx" event={"ID":"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3","Type":"ContainerStarted","Data":"6235a5a575b7fa9fe5cd8a4b10e0decc05bff951f03b03385f6dfa4e8588ac0e"}
Sep 09 03:42:48.161723 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:48.161543    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx" event={"ID":"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3","Type":"ContainerStarted","Data":"2e343c6a4a0bed2ef1566731706a0d2c8468ece84294b9d44fc8aa79d68d69a4"}
Sep 09 03:42:48.161723 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:48.161633    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:42:48.163287 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:48.163258    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c" event={"ID":"d0aff821-b750-4f32-b217-a1b0173b5a73","Type":"ContainerStarted","Data":"8dfc3bb762ddba58f69e497a649e0d8cbf855d92ce0146566f583713b41e1136"}
Sep 09 03:42:48.163402 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:48.163316    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c"
Sep 09 03:42:48.163402 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:48.163340    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c" event={"ID":"d0aff821-b750-4f32-b217-a1b0173b5a73","Type":"ContainerStarted","Data":"1cf7adf4ef829bcc3e85beef98826fce11db8879784b9db839bdfea0422d8067"}
Sep 09 03:42:48.181361 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:48.181314    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx" podStartSLOduration=1.865407643 podStartE2EDuration="2.181298207s" totalImagesPullingTime="315.890564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:42:48.179354999 +0000 UTC m=+1785.316102863" watchObservedRunningTime="2026-09-09 03:42:48.181298207 +0000 UTC m=+1785.318046084"
Sep 09 03:42:48.197611 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:48.197547    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c" podStartSLOduration=1.197527305 podStartE2EDuration="1.197527305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:42:48.19670423 +0000 UTC m=+1785.333452096" watchObservedRunningTime="2026-09-09 03:42:48.197527305 +0000 UTC m=+1785.334275173"
Sep 09 03:42:52.795802 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:52.795768    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:42:54.174122 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:54.174094    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:42:59.170824 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:42:59.170747    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c"
Sep 09 03:43:09.505061 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:09.505035    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:43:09.534293 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:09.534266    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:43:10.524884 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:10.524851    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:43:15.890209 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:15.890167    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c_d0aff821-b750-4f32-b217-a1b0173b5a73/payload-processing/0.log"
Sep 09 03:43:21.113038 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:21.112992    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:43:24.901634 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:24.901582    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:43:25.333520 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:25.333438    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:43:46.197677 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:46.197638    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:43:51.891875 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:51.891843    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c_d0aff821-b750-4f32-b217-a1b0173b5a73/payload-processing/0.log"
Sep 09 03:43:52.183103 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:52.183017    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:43:52.504239 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:52.504153    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:43:52.729907 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:52.729873    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:43:55.834904 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:55.834864    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:43:55.931943 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:55.931910    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:43:56.954876 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:56.954843    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"]
Sep 09 03:43:56.959132 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:56.959106    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:43:56.961807 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:56.961783    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-65a459-serving-cert\""
Sep 09 03:43:56.971348 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:56.971327    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"]
Sep 09 03:43:57.001351 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:57.001321    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:43:57.034938 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:57.034911    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e1e6755-57e9-4c46-a685-d7e689d3d660-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k\" (UID: \"8e1e6755-57e9-4c46-a685-d7e689d3d660\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:43:57.035073 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:57.034962    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e1e6755-57e9-4c46-a685-d7e689d3d660-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k\" (UID: \"8e1e6755-57e9-4c46-a685-d7e689d3d660\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:43:57.035073 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:57.034986    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fk7v\" (UniqueName: \"kubernetes.io/projected/8e1e6755-57e9-4c46-a685-d7e689d3d660-kube-api-access-8fk7v\") pod \"maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k\" (UID: \"8e1e6755-57e9-4c46-a685-d7e689d3d660\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:43:57.035073 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:57.035011    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e6755-57e9-4c46-a685-d7e689d3d660-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k\" (UID: \"8e1e6755-57e9-4c46-a685-d7e689d3d660\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:43:57.104295 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:57.104253    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:43:57.135428 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:57.135396    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e1e6755-57e9-4c46-a685-d7e689d3d660-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k\" (UID: \"8e1e6755-57e9-4c46-a685-d7e689d3d660\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:43:57.135556 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:57.135438    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fk7v\" (UniqueName: \"kubernetes.io/projected/8e1e6755-57e9-4c46-a685-d7e689d3d660-kube-api-access-8fk7v\") pod \"maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k\" (UID: \"8e1e6755-57e9-4c46-a685-d7e689d3d660\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:43:57.135658 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:57.135636    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e6755-57e9-4c46-a685-d7e689d3d660-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k\" (UID: \"8e1e6755-57e9-4c46-a685-d7e689d3d660\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:43:57.135808 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:57.135789    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e1e6755-57e9-4c46-a685-d7e689d3d660-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k\" (UID: \"8e1e6755-57e9-4c46-a685-d7e689d3d660\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:43:57.135930 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:43:57.135914    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-serving-cert: secret "maas-api-e2e-ipp-cleanup-65a459-serving-cert" not found
Sep 09 03:43:57.136002 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:43:57.135994    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e1e6755-57e9-4c46-a685-d7e689d3d660-maas-api-tls podName:8e1e6755-57e9-4c46-a685-d7e689d3d660 nodeName:}" failed. No retries permitted until 2026-09-09 03:43:57.635972769 +0000 UTC m=+1854.772720619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e1e6755-57e9-4c46-a685-d7e689d3d660-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k" (UID: "8e1e6755-57e9-4c46-a685-d7e689d3d660") : secret "maas-api-e2e-ipp-cleanup-65a459-serving-cert" not found
Sep 09 03:43:57.136268 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:57.136248    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e6755-57e9-4c46-a685-d7e689d3d660-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k\" (UID: \"8e1e6755-57e9-4c46-a685-d7e689d3d660\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:43:57.137829 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:57.137812    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e1e6755-57e9-4c46-a685-d7e689d3d660-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k\" (UID: \"8e1e6755-57e9-4c46-a685-d7e689d3d660\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:43:57.144897 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:57.144874    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fk7v\" (UniqueName: \"kubernetes.io/projected/8e1e6755-57e9-4c46-a685-d7e689d3d660-kube-api-access-8fk7v\") pod \"maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k\" (UID: \"8e1e6755-57e9-4c46-a685-d7e689d3d660\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:43:57.463659 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:57.463578    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:43:57.563318 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:57.563288    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:43:57.641845 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:57.641814    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e1e6755-57e9-4c46-a685-d7e689d3d660-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k\" (UID: \"8e1e6755-57e9-4c46-a685-d7e689d3d660\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:43:57.644300 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:57.644269    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e1e6755-57e9-4c46-a685-d7e689d3d660-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k\" (UID: \"8e1e6755-57e9-4c46-a685-d7e689d3d660\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:43:57.873221 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:57.873189    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:43:58.143021 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:58.142991    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"]
Sep 09 03:43:58.148353 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:43:58.148328    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1e6755_57e9_4c46_a685_d7e689d3d660.slice/crio-36bdb8767856ff47977e13458ffdf22c6107c71116122cbc5e39e1d440748974 WatchSource:0}: Error finding container 36bdb8767856ff47977e13458ffdf22c6107c71116122cbc5e39e1d440748974: Status 404 returned error can't find the container with id 36bdb8767856ff47977e13458ffdf22c6107c71116122cbc5e39e1d440748974
Sep 09 03:43:58.520542 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:58.520455    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k" event={"ID":"8e1e6755-57e9-4c46-a685-d7e689d3d660","Type":"ContainerStarted","Data":"36bdb8767856ff47977e13458ffdf22c6107c71116122cbc5e39e1d440748974"}
Sep 09 03:43:59.527280 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:59.527247    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k" event={"ID":"8e1e6755-57e9-4c46-a685-d7e689d3d660","Type":"ContainerStarted","Data":"628c9e2fa2478b50773740102afe93d5cf03ee40191d34555ff9da28657d18ef"}
Sep 09 03:43:59.527682 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:59.527386    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:43:59.546207 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:43:59.546159    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k" podStartSLOduration=3.134040824 podStartE2EDuration="3.546140973s" totalImagesPullingTime="412.100149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:43:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:43:59.544725999 +0000 UTC m=+1856.681473866" watchObservedRunningTime="2026-09-09 03:43:59.546140973 +0000 UTC m=+1856.682888876"
Sep 09 03:44:05.093176 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:05.093135    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:44:05.538742 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:05.538711    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:44:20.921226 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:20.921189    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:44:27.510287 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:27.510201    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"]
Sep 09 03:44:27.510842 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:27.510463    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k" podUID="8e1e6755-57e9-4c46-a685-d7e689d3d660" containerName="maas-api" containerID="cri-o://628c9e2fa2478b50773740102afe93d5cf03ee40191d34555ff9da28657d18ef" gracePeriod=30
Sep 09 03:44:27.556784 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:27.556751    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:44:27.655393 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:27.655359    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:44:27.678065 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:27.678030    2560 generic.go:356] "Generic (PLEG): container finished" podID="8e1e6755-57e9-4c46-a685-d7e689d3d660" containerID="628c9e2fa2478b50773740102afe93d5cf03ee40191d34555ff9da28657d18ef" exitCode=0
Sep 09 03:44:27.678211 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:27.678096    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k" event={"ID":"8e1e6755-57e9-4c46-a685-d7e689d3d660","Type":"ContainerDied","Data":"628c9e2fa2478b50773740102afe93d5cf03ee40191d34555ff9da28657d18ef"}
Sep 09 03:44:27.754133 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:27.754105    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:44:27.760927 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:27.760877    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:44:27.858667 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:27.858636    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:44:27.939342 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:27.939313    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e6755-57e9-4c46-a685-d7e689d3d660-openshift-service-ca\") pod \"8e1e6755-57e9-4c46-a685-d7e689d3d660\" (UID: \"8e1e6755-57e9-4c46-a685-d7e689d3d660\") "
Sep 09 03:44:27.939530 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:27.939389    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fk7v\" (UniqueName: \"kubernetes.io/projected/8e1e6755-57e9-4c46-a685-d7e689d3d660-kube-api-access-8fk7v\") pod \"8e1e6755-57e9-4c46-a685-d7e689d3d660\" (UID: \"8e1e6755-57e9-4c46-a685-d7e689d3d660\") "
Sep 09 03:44:27.939530 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:27.939425    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e1e6755-57e9-4c46-a685-d7e689d3d660-maas-api-tls\") pod \"8e1e6755-57e9-4c46-a685-d7e689d3d660\" (UID: \"8e1e6755-57e9-4c46-a685-d7e689d3d660\") "
Sep 09 03:44:27.939530 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:27.939451    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e1e6755-57e9-4c46-a685-d7e689d3d660-metrics-certs\") pod \"8e1e6755-57e9-4c46-a685-d7e689d3d660\" (UID: \"8e1e6755-57e9-4c46-a685-d7e689d3d660\") "
Sep 09 03:44:27.939769 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:27.939740    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1e6755-57e9-4c46-a685-d7e689d3d660-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8e1e6755-57e9-4c46-a685-d7e689d3d660" (UID: "8e1e6755-57e9-4c46-a685-d7e689d3d660"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:44:27.941573 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:27.941538    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1e6755-57e9-4c46-a685-d7e689d3d660-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8e1e6755-57e9-4c46-a685-d7e689d3d660" (UID: "8e1e6755-57e9-4c46-a685-d7e689d3d660"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:44:27.941573 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:27.941554    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1e6755-57e9-4c46-a685-d7e689d3d660-kube-api-access-8fk7v" (OuterVolumeSpecName: "kube-api-access-8fk7v") pod "8e1e6755-57e9-4c46-a685-d7e689d3d660" (UID: "8e1e6755-57e9-4c46-a685-d7e689d3d660"). InnerVolumeSpecName "kube-api-access-8fk7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:44:27.941748 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:27.941608    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1e6755-57e9-4c46-a685-d7e689d3d660-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8e1e6755-57e9-4c46-a685-d7e689d3d660" (UID: "8e1e6755-57e9-4c46-a685-d7e689d3d660"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:44:28.040715 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:28.040645    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e6755-57e9-4c46-a685-d7e689d3d660-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:44:28.040715 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:28.040665    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fk7v\" (UniqueName: \"kubernetes.io/projected/8e1e6755-57e9-4c46-a685-d7e689d3d660-kube-api-access-8fk7v\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:44:28.040715 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:28.040675    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e1e6755-57e9-4c46-a685-d7e689d3d660-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:44:28.040715 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:28.040683    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e1e6755-57e9-4c46-a685-d7e689d3d660-metrics-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:44:28.165443 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:28.165403    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:44:28.684520 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:28.684489    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"
Sep 09 03:44:28.684991 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:28.684488    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k" event={"ID":"8e1e6755-57e9-4c46-a685-d7e689d3d660","Type":"ContainerDied","Data":"36bdb8767856ff47977e13458ffdf22c6107c71116122cbc5e39e1d440748974"}
Sep 09 03:44:28.684991 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:28.684662    2560 scope.go:117] "RemoveContainer" containerID="628c9e2fa2478b50773740102afe93d5cf03ee40191d34555ff9da28657d18ef"
Sep 09 03:44:28.707200 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:28.707177    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"]
Sep 09 03:44:28.711572 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:28.711550    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65a459-8c94b8f65-99l4k"]
Sep 09 03:44:29.358654 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:29.358620    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e1e6755-57e9-4c46-a685-d7e689d3d660" path="/var/lib/kubelet/pods/8e1e6755-57e9-4c46-a685-d7e689d3d660/volumes"
Sep 09 03:44:31.805062 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:31.805020    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:44:34.156478 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:34.156441    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:44:34.237655 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:34.237589    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:44:34.443093 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:34.443000    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:44:38.321998 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:38.321964    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:44:48.840642 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:48.840580    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:44:52.389100 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:52.389066    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:44:54.527414 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.527375    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"]
Sep 09 03:44:54.527844 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.527613    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w" podUID="b5bd6fff-fb27-4a76-a230-9d13b4cb74a2" containerName="maas-api" containerID="cri-o://0fddacfe486a73a9ad05892ba8638e6321a80995f9fadb8981dec46209e7dc37" gracePeriod=30
Sep 09 03:44:54.559822 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.559790    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:44:54.655399 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.655371    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:44:54.750871 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.750841    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:44:54.777856 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.777792    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:44:54.821975 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.821942    2560 generic.go:356] "Generic (PLEG): container finished" podID="b5bd6fff-fb27-4a76-a230-9d13b4cb74a2" containerID="0fddacfe486a73a9ad05892ba8638e6321a80995f9fadb8981dec46209e7dc37" exitCode=0
Sep 09 03:44:54.822144 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.821999    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"
Sep 09 03:44:54.822144 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.822019    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w" event={"ID":"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2","Type":"ContainerDied","Data":"0fddacfe486a73a9ad05892ba8638e6321a80995f9fadb8981dec46209e7dc37"}
Sep 09 03:44:54.822144 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.822072    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w" event={"ID":"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2","Type":"ContainerDied","Data":"d09f49d446ab8b705f2fd8ab0e59af1165081c4fe2d02c296919c0edae20bee9"}
Sep 09 03:44:54.822144 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.822101    2560 scope.go:117] "RemoveContainer" containerID="0fddacfe486a73a9ad05892ba8638e6321a80995f9fadb8981dec46209e7dc37"
Sep 09 03:44:54.831791 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.831769    2560 scope.go:117] "RemoveContainer" containerID="0fddacfe486a73a9ad05892ba8638e6321a80995f9fadb8981dec46209e7dc37"
Sep 09 03:44:54.832086 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:44:54.832065    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fddacfe486a73a9ad05892ba8638e6321a80995f9fadb8981dec46209e7dc37\": container with ID starting with 0fddacfe486a73a9ad05892ba8638e6321a80995f9fadb8981dec46209e7dc37 not found: ID does not exist" containerID="0fddacfe486a73a9ad05892ba8638e6321a80995f9fadb8981dec46209e7dc37"
Sep 09 03:44:54.832165 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.832097    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fddacfe486a73a9ad05892ba8638e6321a80995f9fadb8981dec46209e7dc37"} err="failed to get container status \"0fddacfe486a73a9ad05892ba8638e6321a80995f9fadb8981dec46209e7dc37\": rpc error: code = NotFound desc = could not find container \"0fddacfe486a73a9ad05892ba8638e6321a80995f9fadb8981dec46209e7dc37\": container with ID starting with 0fddacfe486a73a9ad05892ba8638e6321a80995f9fadb8981dec46209e7dc37 not found: ID does not exist"
Sep 09 03:44:54.850047 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.850019    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:44:54.891458 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.891429    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-maas-api-tls\") pod \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\" (UID: \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\") "
Sep 09 03:44:54.891571 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.891472    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xq8vq\" (UniqueName: \"kubernetes.io/projected/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-kube-api-access-xq8vq\") pod \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\" (UID: \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\") "
Sep 09 03:44:54.891571 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.891520    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-metrics-certs\") pod \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\" (UID: \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\") "
Sep 09 03:44:54.891571 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.891553    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-openshift-service-ca\") pod \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\" (UID: \"b5bd6fff-fb27-4a76-a230-9d13b4cb74a2\") "
Sep 09 03:44:54.891991 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.891963    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b5bd6fff-fb27-4a76-a230-9d13b4cb74a2" (UID: "b5bd6fff-fb27-4a76-a230-9d13b4cb74a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:44:54.893684 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.893657    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b5bd6fff-fb27-4a76-a230-9d13b4cb74a2" (UID: "b5bd6fff-fb27-4a76-a230-9d13b4cb74a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:44:54.893684 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.893672    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b5bd6fff-fb27-4a76-a230-9d13b4cb74a2" (UID: "b5bd6fff-fb27-4a76-a230-9d13b4cb74a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:44:54.893799 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.893696    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-kube-api-access-xq8vq" (OuterVolumeSpecName: "kube-api-access-xq8vq") pod "b5bd6fff-fb27-4a76-a230-9d13b4cb74a2" (UID: "b5bd6fff-fb27-4a76-a230-9d13b4cb74a2"). InnerVolumeSpecName "kube-api-access-xq8vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:44:54.993456 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.993428    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:44:54.993456 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.993455    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq8vq\" (UniqueName: \"kubernetes.io/projected/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-kube-api-access-xq8vq\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:44:54.993644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.993466    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-metrics-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:44:54.993644 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:54.993474    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:44:55.144944 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:55.144912    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"]
Sep 09 03:44:55.149868 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:55.149843    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-46be70-5ff9dc59b8-cwg5w"]
Sep 09 03:44:55.323815 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:55.323758    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:44:55.359042 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:44:55.359015    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5bd6fff-fb27-4a76-a230-9d13b4cb74a2" path="/var/lib/kubelet/pods/b5bd6fff-fb27-4a76-a230-9d13b4cb74a2/volumes"
Sep 09 03:45:00.345965 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:00.345932    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:45:01.606377 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:01.606331    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:45:03.700489 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:03.700457    2560 scope.go:117] "RemoveContainer" containerID="8a5eed84b5d130782b5f30aaed56117dc74ae4893066d0b4b57117625bb6c027"
Sep 09 03:45:21.794259 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:21.794226    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:45:36.568329 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:36.568288    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:45:37.771681 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:37.771642    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:45:43.381395 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:43.381361    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:45:43.481731 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:43.481695    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:45:43.723349 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:43.723254    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:45:44.387930 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:44.387897    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:45:54.632284 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.632251    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"]
Sep 09 03:45:54.632726 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.632560    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx" podUID="fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3" containerName="maas-api" containerID="cri-o://6235a5a575b7fa9fe5cd8a4b10e0decc05bff951f03b03385f6dfa4e8588ac0e" gracePeriod=30
Sep 09 03:45:54.651745 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.651714    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:45:54.678718 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.678686    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c"]
Sep 09 03:45:54.678932 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.678900    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c" podUID="d0aff821-b750-4f32-b217-a1b0173b5a73" containerName="payload-processing" containerID="cri-o://8dfc3bb762ddba58f69e497a649e0d8cbf855d92ce0146566f583713b41e1136" gracePeriod=30
Sep 09 03:45:54.750870 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.750830    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:45:54.853933 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.853895    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:45:54.920087 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.920063    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:45:54.943798 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.943776    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:45:54.946475 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.946455    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c"
Sep 09 03:45:54.954385 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.954363    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-maas-api-tls\") pod \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\" (UID: \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\") "
Sep 09 03:45:54.954513 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.954398    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-metrics-certs\") pod \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\" (UID: \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\") "
Sep 09 03:45:54.954513 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.954424    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0aff821-b750-4f32-b217-a1b0173b5a73-plugins-config-volume\") pod \"d0aff821-b750-4f32-b217-a1b0173b5a73\" (UID: \"d0aff821-b750-4f32-b217-a1b0173b5a73\") "
Sep 09 03:45:54.954513 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.954482    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-openshift-service-ca\") pod \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\" (UID: \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\") "
Sep 09 03:45:54.954513 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.954500    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9g9c\" (UniqueName: \"kubernetes.io/projected/d0aff821-b750-4f32-b217-a1b0173b5a73-kube-api-access-c9g9c\") pod \"d0aff821-b750-4f32-b217-a1b0173b5a73\" (UID: \"d0aff821-b750-4f32-b217-a1b0173b5a73\") "
Sep 09 03:45:54.954740 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.954560    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kznmd\" (UniqueName: \"kubernetes.io/projected/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-kube-api-access-kznmd\") pod \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\" (UID: \"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3\") "
Sep 09 03:45:54.954890 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.954855    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0aff821-b750-4f32-b217-a1b0173b5a73-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0aff821-b750-4f32-b217-a1b0173b5a73" (UID: "d0aff821-b750-4f32-b217-a1b0173b5a73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:45:54.955334 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.955147    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0aff821-b750-4f32-b217-a1b0173b5a73-plugins-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:45:54.955334 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.955244    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3" (UID: "fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:45:54.957496 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.957472    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3" (UID: "fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:45:54.957496 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.957486    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3" (UID: "fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:45:54.957708 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.957688    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0aff821-b750-4f32-b217-a1b0173b5a73-kube-api-access-c9g9c" (OuterVolumeSpecName: "kube-api-access-c9g9c") pod "d0aff821-b750-4f32-b217-a1b0173b5a73" (UID: "d0aff821-b750-4f32-b217-a1b0173b5a73"). InnerVolumeSpecName "kube-api-access-c9g9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:45:54.957891 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:54.957876    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-kube-api-access-kznmd" (OuterVolumeSpecName: "kube-api-access-kznmd") pod "fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3" (UID: "fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3"). InnerVolumeSpecName "kube-api-access-kznmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:45:55.055623 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.055563    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:45:55.055623 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.055618    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-metrics-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:45:55.055623 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.055631    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:45:55.055866 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.055640    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9g9c\" (UniqueName: \"kubernetes.io/projected/d0aff821-b750-4f32-b217-a1b0173b5a73-kube-api-access-c9g9c\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:45:55.055866 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.055649    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kznmd\" (UniqueName: \"kubernetes.io/projected/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3-kube-api-access-kznmd\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:45:55.126114 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.126081    2560 generic.go:356] "Generic (PLEG): container finished" podID="d0aff821-b750-4f32-b217-a1b0173b5a73" containerID="8dfc3bb762ddba58f69e497a649e0d8cbf855d92ce0146566f583713b41e1136" exitCode=0
Sep 09 03:45:55.126293 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.126126    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c"
Sep 09 03:45:55.126293 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.126170    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c" event={"ID":"d0aff821-b750-4f32-b217-a1b0173b5a73","Type":"ContainerDied","Data":"8dfc3bb762ddba58f69e497a649e0d8cbf855d92ce0146566f583713b41e1136"}
Sep 09 03:45:55.126293 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.126225    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c" event={"ID":"d0aff821-b750-4f32-b217-a1b0173b5a73","Type":"ContainerDied","Data":"1cf7adf4ef829bcc3e85beef98826fce11db8879784b9db839bdfea0422d8067"}
Sep 09 03:45:55.126293 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.126256    2560 scope.go:117] "RemoveContainer" containerID="8dfc3bb762ddba58f69e497a649e0d8cbf855d92ce0146566f583713b41e1136"
Sep 09 03:45:55.127708 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.127683    2560 generic.go:356] "Generic (PLEG): container finished" podID="fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3" containerID="6235a5a575b7fa9fe5cd8a4b10e0decc05bff951f03b03385f6dfa4e8588ac0e" exitCode=0
Sep 09 03:45:55.127832 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.127724    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"
Sep 09 03:45:55.127832 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.127747    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx" event={"ID":"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3","Type":"ContainerDied","Data":"6235a5a575b7fa9fe5cd8a4b10e0decc05bff951f03b03385f6dfa4e8588ac0e"}
Sep 09 03:45:55.127832 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.127788    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx" event={"ID":"fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3","Type":"ContainerDied","Data":"2e343c6a4a0bed2ef1566731706a0d2c8468ece84294b9d44fc8aa79d68d69a4"}
Sep 09 03:45:55.136890 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.136874    2560 scope.go:117] "RemoveContainer" containerID="8dfc3bb762ddba58f69e497a649e0d8cbf855d92ce0146566f583713b41e1136"
Sep 09 03:45:55.137241 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:45:55.137217    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dfc3bb762ddba58f69e497a649e0d8cbf855d92ce0146566f583713b41e1136\": container with ID starting with 8dfc3bb762ddba58f69e497a649e0d8cbf855d92ce0146566f583713b41e1136 not found: ID does not exist" containerID="8dfc3bb762ddba58f69e497a649e0d8cbf855d92ce0146566f583713b41e1136"
Sep 09 03:45:55.137289 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.137256    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dfc3bb762ddba58f69e497a649e0d8cbf855d92ce0146566f583713b41e1136"} err="failed to get container status \"8dfc3bb762ddba58f69e497a649e0d8cbf855d92ce0146566f583713b41e1136\": rpc error: code = NotFound desc = could not find container \"8dfc3bb762ddba58f69e497a649e0d8cbf855d92ce0146566f583713b41e1136\": container with ID starting with 8dfc3bb762ddba58f69e497a649e0d8cbf855d92ce0146566f583713b41e1136 not found: ID does not exist"
Sep 09 03:45:55.137289 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.137279    2560 scope.go:117] "RemoveContainer" containerID="6235a5a575b7fa9fe5cd8a4b10e0decc05bff951f03b03385f6dfa4e8588ac0e"
Sep 09 03:45:55.145513 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.145493    2560 scope.go:117] "RemoveContainer" containerID="6235a5a575b7fa9fe5cd8a4b10e0decc05bff951f03b03385f6dfa4e8588ac0e"
Sep 09 03:45:55.145785 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:45:55.145764    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6235a5a575b7fa9fe5cd8a4b10e0decc05bff951f03b03385f6dfa4e8588ac0e\": container with ID starting with 6235a5a575b7fa9fe5cd8a4b10e0decc05bff951f03b03385f6dfa4e8588ac0e not found: ID does not exist" containerID="6235a5a575b7fa9fe5cd8a4b10e0decc05bff951f03b03385f6dfa4e8588ac0e"
Sep 09 03:45:55.145835 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.145793    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6235a5a575b7fa9fe5cd8a4b10e0decc05bff951f03b03385f6dfa4e8588ac0e"} err="failed to get container status \"6235a5a575b7fa9fe5cd8a4b10e0decc05bff951f03b03385f6dfa4e8588ac0e\": rpc error: code = NotFound desc = could not find container \"6235a5a575b7fa9fe5cd8a4b10e0decc05bff951f03b03385f6dfa4e8588ac0e\": container with ID starting with 6235a5a575b7fa9fe5cd8a4b10e0decc05bff951f03b03385f6dfa4e8588ac0e not found: ID does not exist"
Sep 09 03:45:55.148855 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.148836    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c"]
Sep 09 03:45:55.152745 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.152722    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60a731-744df5f759-pj72c"]
Sep 09 03:45:55.167153 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.167124    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"]
Sep 09 03:45:55.172939 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.172914    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60a731-7479bcb7f5-lrdvx"]
Sep 09 03:45:55.267525 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.267488    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:45:55.361406 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.361371    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0aff821-b750-4f32-b217-a1b0173b5a73" path="/var/lib/kubelet/pods/d0aff821-b750-4f32-b217-a1b0173b5a73/volumes"
Sep 09 03:45:55.361836 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:55.361820    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3" path="/var/lib/kubelet/pods/fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3/volumes"
Sep 09 03:45:57.685407 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:45:57.685372    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:46:01.188315 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:01.188276    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:01.299325 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:01.299284    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:01.482028 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:01.481937    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:02.062481 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:02.062447    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:03.728219 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:03.728187    2560 scope.go:117] "RemoveContainer" containerID="96306c8f0672e8c8479d6e71f2c2220f80aef9c38c457059707cd33d60ad6910"
Sep 09 03:46:03.736519 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:03.736501    2560 scope.go:117] "RemoveContainer" containerID="1ca6b389b8ec7886e08118b10e2d3a0a8b2c28f6bb5cde058e0cd4f1e8c17203"
Sep 09 03:46:12.451033 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.450998    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"]
Sep 09 03:46:12.451566 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.451251    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm" podUID="b4ea9840-b602-4676-a8f2-122a5403a336" containerName="maas-api" containerID="cri-o://b6910d441c4a465f02c12a3fbe9515b15c381fd2bfb48b78197d3c7d8dcbe43a" gracePeriod=30
Sep 09 03:46:12.466998 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.466962    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm" podUID="b4ea9840-b602-4676-a8f2-122a5403a336" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.62:8443/health\": dial tcp 10.132.0.62:8443: connect: connection refused"
Sep 09 03:46:12.481622 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.481572    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:12.545774 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.545735    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:12.649396 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.649364    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:12.704307 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.704251    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:46:12.719058 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.719019    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:12.823282 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.823251    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nx257\" (UniqueName: \"kubernetes.io/projected/b4ea9840-b602-4676-a8f2-122a5403a336-kube-api-access-nx257\") pod \"b4ea9840-b602-4676-a8f2-122a5403a336\" (UID: \"b4ea9840-b602-4676-a8f2-122a5403a336\") "
Sep 09 03:46:12.823441 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.823345    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea9840-b602-4676-a8f2-122a5403a336-metrics-certs\") pod \"b4ea9840-b602-4676-a8f2-122a5403a336\" (UID: \"b4ea9840-b602-4676-a8f2-122a5403a336\") "
Sep 09 03:46:12.823441 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.823364    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ea9840-b602-4676-a8f2-122a5403a336-openshift-service-ca\") pod \"b4ea9840-b602-4676-a8f2-122a5403a336\" (UID: \"b4ea9840-b602-4676-a8f2-122a5403a336\") "
Sep 09 03:46:12.823441 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.823382    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4ea9840-b602-4676-a8f2-122a5403a336-maas-api-tls\") pod \"b4ea9840-b602-4676-a8f2-122a5403a336\" (UID: \"b4ea9840-b602-4676-a8f2-122a5403a336\") "
Sep 09 03:46:12.823781 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.823751    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ea9840-b602-4676-a8f2-122a5403a336-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b4ea9840-b602-4676-a8f2-122a5403a336" (UID: "b4ea9840-b602-4676-a8f2-122a5403a336"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:46:12.825554 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.825525    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4ea9840-b602-4676-a8f2-122a5403a336-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b4ea9840-b602-4676-a8f2-122a5403a336" (UID: "b4ea9840-b602-4676-a8f2-122a5403a336"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:46:12.825665 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.825577    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ea9840-b602-4676-a8f2-122a5403a336-kube-api-access-nx257" (OuterVolumeSpecName: "kube-api-access-nx257") pod "b4ea9840-b602-4676-a8f2-122a5403a336" (UID: "b4ea9840-b602-4676-a8f2-122a5403a336"). InnerVolumeSpecName "kube-api-access-nx257". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:46:12.825665 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.825568    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4ea9840-b602-4676-a8f2-122a5403a336-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b4ea9840-b602-4676-a8f2-122a5403a336" (UID: "b4ea9840-b602-4676-a8f2-122a5403a336"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:46:12.924658 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.924578    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx257\" (UniqueName: \"kubernetes.io/projected/b4ea9840-b602-4676-a8f2-122a5403a336-kube-api-access-nx257\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:46:12.924658 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.924652    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea9840-b602-4676-a8f2-122a5403a336-metrics-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:46:12.924658 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.924662    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ea9840-b602-4676-a8f2-122a5403a336-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:46:12.924658 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:12.924670    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4ea9840-b602-4676-a8f2-122a5403a336-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:46:13.089732 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:13.089630    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:13.226038 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:13.226006    2560 generic.go:356] "Generic (PLEG): container finished" podID="b4ea9840-b602-4676-a8f2-122a5403a336" containerID="b6910d441c4a465f02c12a3fbe9515b15c381fd2bfb48b78197d3c7d8dcbe43a" exitCode=0
Sep 09 03:46:13.226212 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:13.226047    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"
Sep 09 03:46:13.226212 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:13.226058    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm" event={"ID":"b4ea9840-b602-4676-a8f2-122a5403a336","Type":"ContainerDied","Data":"b6910d441c4a465f02c12a3fbe9515b15c381fd2bfb48b78197d3c7d8dcbe43a"}
Sep 09 03:46:13.226212 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:13.226102    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm" event={"ID":"b4ea9840-b602-4676-a8f2-122a5403a336","Type":"ContainerDied","Data":"5c6dc55a09f5d036d96c60d80a54843d0bef49ffef78c9e1530485c7b3098571"}
Sep 09 03:46:13.226212 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:13.226123    2560 scope.go:117] "RemoveContainer" containerID="b6910d441c4a465f02c12a3fbe9515b15c381fd2bfb48b78197d3c7d8dcbe43a"
Sep 09 03:46:13.235423 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:13.235406    2560 scope.go:117] "RemoveContainer" containerID="b6910d441c4a465f02c12a3fbe9515b15c381fd2bfb48b78197d3c7d8dcbe43a"
Sep 09 03:46:13.235727 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:46:13.235706    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6910d441c4a465f02c12a3fbe9515b15c381fd2bfb48b78197d3c7d8dcbe43a\": container with ID starting with b6910d441c4a465f02c12a3fbe9515b15c381fd2bfb48b78197d3c7d8dcbe43a not found: ID does not exist" containerID="b6910d441c4a465f02c12a3fbe9515b15c381fd2bfb48b78197d3c7d8dcbe43a"
Sep 09 03:46:13.235809 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:13.235736    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6910d441c4a465f02c12a3fbe9515b15c381fd2bfb48b78197d3c7d8dcbe43a"} err="failed to get container status \"b6910d441c4a465f02c12a3fbe9515b15c381fd2bfb48b78197d3c7d8dcbe43a\": rpc error: code = NotFound desc = could not find container \"b6910d441c4a465f02c12a3fbe9515b15c381fd2bfb48b78197d3c7d8dcbe43a\": container with ID starting with b6910d441c4a465f02c12a3fbe9515b15c381fd2bfb48b78197d3c7d8dcbe43a not found: ID does not exist"
Sep 09 03:46:13.248062 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:13.248041    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"]
Sep 09 03:46:13.252126 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:13.252106    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5d8e85-54d545dd46-t4zmm"]
Sep 09 03:46:13.358814 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:13.358786    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4ea9840-b602-4676-a8f2-122a5403a336" path="/var/lib/kubelet/pods/b4ea9840-b602-4676-a8f2-122a5403a336/volumes"
Sep 09 03:46:15.392978 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:15.392942    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:46:19.123214 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:19.123179    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:19.195865 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:19.195833    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:19.410352 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:19.410271    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:19.510276 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:19.510238    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:20.319253 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:20.319215    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:30.285760 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.285718    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"]
Sep 09 03:46:30.286238 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.285933    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc" podUID="d710a0a8-d44b-41b0-865a-458e9fd88c4f" containerName="maas-api" containerID="cri-o://6cbdcc3822e54c41c0526d2945018d903986a7e0318c6ba6ac5983e9d8d367c1" gracePeriod=30
Sep 09 03:46:30.321328 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.321293    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:30.399121 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.399087    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:30.475308 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.475263    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:30.529108 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.529082    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:46:30.568244 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.568164    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:30.585782 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.585751    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d710a0a8-d44b-41b0-865a-458e9fd88c4f-openshift-service-ca\") pod \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\" (UID: \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\") "
Sep 09 03:46:30.585923 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.585844    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d710a0a8-d44b-41b0-865a-458e9fd88c4f-metrics-certs\") pod \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\" (UID: \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\") "
Sep 09 03:46:30.585923 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.585870    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwfw4\" (UniqueName: \"kubernetes.io/projected/d710a0a8-d44b-41b0-865a-458e9fd88c4f-kube-api-access-zwfw4\") pod \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\" (UID: \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\") "
Sep 09 03:46:30.585923 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.585895    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d710a0a8-d44b-41b0-865a-458e9fd88c4f-maas-api-tls\") pod \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\" (UID: \"d710a0a8-d44b-41b0-865a-458e9fd88c4f\") "
Sep 09 03:46:30.586155 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.586130    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d710a0a8-d44b-41b0-865a-458e9fd88c4f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d710a0a8-d44b-41b0-865a-458e9fd88c4f" (UID: "d710a0a8-d44b-41b0-865a-458e9fd88c4f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:46:30.587995 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.587974    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d710a0a8-d44b-41b0-865a-458e9fd88c4f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d710a0a8-d44b-41b0-865a-458e9fd88c4f" (UID: "d710a0a8-d44b-41b0-865a-458e9fd88c4f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:46:30.588070 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.588002    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d710a0a8-d44b-41b0-865a-458e9fd88c4f-kube-api-access-zwfw4" (OuterVolumeSpecName: "kube-api-access-zwfw4") pod "d710a0a8-d44b-41b0-865a-458e9fd88c4f" (UID: "d710a0a8-d44b-41b0-865a-458e9fd88c4f"). InnerVolumeSpecName "kube-api-access-zwfw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:46:30.588112 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.588068    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d710a0a8-d44b-41b0-865a-458e9fd88c4f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d710a0a8-d44b-41b0-865a-458e9fd88c4f" (UID: "d710a0a8-d44b-41b0-865a-458e9fd88c4f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:46:30.687173 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.687138    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d710a0a8-d44b-41b0-865a-458e9fd88c4f-metrics-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:46:30.687173 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.687168    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwfw4\" (UniqueName: \"kubernetes.io/projected/d710a0a8-d44b-41b0-865a-458e9fd88c4f-kube-api-access-zwfw4\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:46:30.687173 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.687179    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d710a0a8-d44b-41b0-865a-458e9fd88c4f-maas-api-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:46:30.687393 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.687188    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d710a0a8-d44b-41b0-865a-458e9fd88c4f-openshift-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:46:30.956987 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:30.956951    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:31.315364 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:31.315274    2560 generic.go:356] "Generic (PLEG): container finished" podID="d710a0a8-d44b-41b0-865a-458e9fd88c4f" containerID="6cbdcc3822e54c41c0526d2945018d903986a7e0318c6ba6ac5983e9d8d367c1" exitCode=0
Sep 09 03:46:31.315364 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:31.315319    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"
Sep 09 03:46:31.315364 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:31.315343    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc" event={"ID":"d710a0a8-d44b-41b0-865a-458e9fd88c4f","Type":"ContainerDied","Data":"6cbdcc3822e54c41c0526d2945018d903986a7e0318c6ba6ac5983e9d8d367c1"}
Sep 09 03:46:31.315910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:31.315385    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc" event={"ID":"d710a0a8-d44b-41b0-865a-458e9fd88c4f","Type":"ContainerDied","Data":"749cc07dea21aae8a9221796cc788955183dd107f8b8352961e5be33f263e2ad"}
Sep 09 03:46:31.315910 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:31.315406    2560 scope.go:117] "RemoveContainer" containerID="6cbdcc3822e54c41c0526d2945018d903986a7e0318c6ba6ac5983e9d8d367c1"
Sep 09 03:46:31.324892 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:31.324873    2560 scope.go:117] "RemoveContainer" containerID="6cbdcc3822e54c41c0526d2945018d903986a7e0318c6ba6ac5983e9d8d367c1"
Sep 09 03:46:31.325187 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:46:31.325164    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cbdcc3822e54c41c0526d2945018d903986a7e0318c6ba6ac5983e9d8d367c1\": container with ID starting with 6cbdcc3822e54c41c0526d2945018d903986a7e0318c6ba6ac5983e9d8d367c1 not found: ID does not exist" containerID="6cbdcc3822e54c41c0526d2945018d903986a7e0318c6ba6ac5983e9d8d367c1"
Sep 09 03:46:31.325264 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:31.325200    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbdcc3822e54c41c0526d2945018d903986a7e0318c6ba6ac5983e9d8d367c1"} err="failed to get container status \"6cbdcc3822e54c41c0526d2945018d903986a7e0318c6ba6ac5983e9d8d367c1\": rpc error: code = NotFound desc = could not find container \"6cbdcc3822e54c41c0526d2945018d903986a7e0318c6ba6ac5983e9d8d367c1\": container with ID starting with 6cbdcc3822e54c41c0526d2945018d903986a7e0318c6ba6ac5983e9d8d367c1 not found: ID does not exist"
Sep 09 03:46:31.337639 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:31.337612    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"]
Sep 09 03:46:31.341682 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:31.341660    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c34148-654bc85d49-zz5tc"]
Sep 09 03:46:31.358820 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:31.358789    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d710a0a8-d44b-41b0-865a-458e9fd88c4f" path="/var/lib/kubelet/pods/d710a0a8-d44b-41b0-865a-458e9fd88c4f/volumes"
Sep 09 03:46:32.889856 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:32.889822    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:46:37.092569 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:37.092523    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:37.190339 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:37.190307    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:37.362411 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:37.362361    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:46:41.486692 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:46:41.486653    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:47:11.199080 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:47:11.199041    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:47:22.297253 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:47:22.297211    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:47:22.397372 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:47:22.397319    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:47:58.264448 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:47:58.264412    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:47:58.350069 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:47:58.350031    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:47:59.859266 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:47:59.859226    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:48:00.987577 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:48:00.987537    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:48:01.904248 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:48:01.904212    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:48:03.161606 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:48:03.161550    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:48:04.624267 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:48:04.624230    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:48:09.539700 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:48:09.539670    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:48:09.563841 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:48:09.563812    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:48:19.059426 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:48:19.059388    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:48:29.181338 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:48:29.181304    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:48:37.198932 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:48:37.198892    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:48:45.089209 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:48:45.089170    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:49:03.102444 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:49:03.102354    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:49:09.493934 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:49:09.493900    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:50:35.328661 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:50:35.328565    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:50:41.295460 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:50:41.295424    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:50:48.997768 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:50:48.997734    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:51:06.000287 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:06.000252    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:51:08.199246 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:08.199209    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:51:19.088082 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088048    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"]
Sep 09 03:51:19.088547 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088401    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3" containerName="maas-api"
Sep 09 03:51:19.088547 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088412    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3" containerName="maas-api"
Sep 09 03:51:19.088547 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088437    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5bd6fff-fb27-4a76-a230-9d13b4cb74a2" containerName="maas-api"
Sep 09 03:51:19.088547 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088445    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5bd6fff-fb27-4a76-a230-9d13b4cb74a2" containerName="maas-api"
Sep 09 03:51:19.088547 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088454    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e1e6755-57e9-4c46-a685-d7e689d3d660" containerName="maas-api"
Sep 09 03:51:19.088547 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088459    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e1e6755-57e9-4c46-a685-d7e689d3d660" containerName="maas-api"
Sep 09 03:51:19.088547 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088469    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4ea9840-b602-4676-a8f2-122a5403a336" containerName="maas-api"
Sep 09 03:51:19.088547 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088475    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4ea9840-b602-4676-a8f2-122a5403a336" containerName="maas-api"
Sep 09 03:51:19.088547 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088484    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0aff821-b750-4f32-b217-a1b0173b5a73" containerName="payload-processing"
Sep 09 03:51:19.088547 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088490    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0aff821-b750-4f32-b217-a1b0173b5a73" containerName="payload-processing"
Sep 09 03:51:19.088547 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088501    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d710a0a8-d44b-41b0-865a-458e9fd88c4f" containerName="maas-api"
Sep 09 03:51:19.088547 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088506    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d710a0a8-d44b-41b0-865a-458e9fd88c4f" containerName="maas-api"
Sep 09 03:51:19.089217 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088561    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc0ca5ad-fc83-4bb8-b821-2650fb0dfff3" containerName="maas-api"
Sep 09 03:51:19.089217 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088569    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4ea9840-b602-4676-a8f2-122a5403a336" containerName="maas-api"
Sep 09 03:51:19.089217 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088575    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0aff821-b750-4f32-b217-a1b0173b5a73" containerName="payload-processing"
Sep 09 03:51:19.089217 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088581    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5bd6fff-fb27-4a76-a230-9d13b4cb74a2" containerName="maas-api"
Sep 09 03:51:19.089217 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088589    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e1e6755-57e9-4c46-a685-d7e689d3d660" containerName="maas-api"
Sep 09 03:51:19.089217 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.088612    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d710a0a8-d44b-41b0-865a-458e9fd88c4f" containerName="maas-api"
Sep 09 03:51:19.091665 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.091644    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.095182 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.095162    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c5827fc9-kserve-self-signed-certs\""
Sep 09 03:51:19.105928 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.105905    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"]
Sep 09 03:51:19.179996 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.179962    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-model-cache\") pod \"e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") " pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.180157 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.180004    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-tmp-dir\") pod \"e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") " pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.180157 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.180127    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/261cdef7-7113-464d-8c37-7eef4ef231c1-tls-certs\") pod \"e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") " pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.180157 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.180152    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-home\") pod \"e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") " pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.180268 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.180207    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m89x8\" (UniqueName: \"kubernetes.io/projected/261cdef7-7113-464d-8c37-7eef4ef231c1-kube-api-access-m89x8\") pod \"e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") " pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.180268 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.180232    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-dshm\") pod \"e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") " pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.191409 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.191379    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:51:19.281654 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.281590    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-dshm\") pod \"e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") " pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.281825 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.281679    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-model-cache\") pod \"e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") " pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.281825 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.281708    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-tmp-dir\") pod \"e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") " pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.281825 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.281797    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/261cdef7-7113-464d-8c37-7eef4ef231c1-tls-certs\") pod \"e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") " pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.281825 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.281823    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-home\") pod \"e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") " pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.282055 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.281880    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m89x8\" (UniqueName: \"kubernetes.io/projected/261cdef7-7113-464d-8c37-7eef4ef231c1-kube-api-access-m89x8\") pod \"e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") " pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.282119 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.282082    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-model-cache\") pod \"e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") " pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.282170 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.282135    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-tmp-dir\") pod \"e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") " pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.282170 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.282150    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-home\") pod \"e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") " pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.283866 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.283845    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-dshm\") pod \"e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") " pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.284174 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.284156    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/261cdef7-7113-464d-8c37-7eef4ef231c1-tls-certs\") pod \"e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") " pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.293957 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.293937    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m89x8\" (UniqueName: \"kubernetes.io/projected/261cdef7-7113-464d-8c37-7eef4ef231c1-kube-api-access-m89x8\") pod \"e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") " pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.401031 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.401005    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:19.671908 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.671882    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"]
Sep 09 03:51:19.674172 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:51:19.674144    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod261cdef7_7113_464d_8c37_7eef4ef231c1.slice/crio-8712b121813292130a86b04096013542bcccefc3983a6ca46b48aabee6c1fa79 WatchSource:0}: Error finding container 8712b121813292130a86b04096013542bcccefc3983a6ca46b48aabee6c1fa79: Status 404 returned error can't find the container with id 8712b121813292130a86b04096013542bcccefc3983a6ca46b48aabee6c1fa79
Sep 09 03:51:19.707541 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:19.707514    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx" event={"ID":"261cdef7-7113-464d-8c37-7eef4ef231c1","Type":"ContainerStarted","Data":"8712b121813292130a86b04096013542bcccefc3983a6ca46b48aabee6c1fa79"}
Sep 09 03:51:20.714043 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:20.714008    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx" event={"ID":"261cdef7-7113-464d-8c37-7eef4ef231c1","Type":"ContainerStarted","Data":"b85e4d1f2aa1ccf8438e52822b9189a14d6404043ad4bccae467d375d55a4f65"}
Sep 09 03:51:20.732222 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:20.732172    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx" podStartSLOduration=1.732156764 podStartE2EDuration="1.732156764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:51:20.73126812 +0000 UTC m=+2297.868015986" watchObservedRunningTime="2026-09-09 03:51:20.732156764 +0000 UTC m=+2297.868904629"
Sep 09 03:51:21.984890 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:21.984857    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:51:24.882666 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:24.882627    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:51:29.401233 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:29.401204    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:29.401742 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:29.401453    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:29.413419 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:29.413395    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:29.768367 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:29.768277    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:51:50.651927 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:50.651894    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:51:51.278739 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:51.278707    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"]
Sep 09 03:51:51.308310 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:51.308279    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:51:53.686183 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:51:53.686143    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:52:00.908162 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:00.908130    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:52:06.199928 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.199890    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:52:06.297925 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.297869    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx" podUID="261cdef7-7113-464d-8c37-7eef4ef231c1" containerName="main" containerID="cri-o://b85e4d1f2aa1ccf8438e52822b9189a14d6404043ad4bccae467d375d55a4f65" gracePeriod=45
Sep 09 03:52:06.531542 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.531513    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:52:06.605462 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.605427    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-home\") pod \"261cdef7-7113-464d-8c37-7eef4ef231c1\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") "
Sep 09 03:52:06.605647 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.605514    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-tmp-dir\") pod \"261cdef7-7113-464d-8c37-7eef4ef231c1\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") "
Sep 09 03:52:06.605647 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.605549    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-dshm\") pod \"261cdef7-7113-464d-8c37-7eef4ef231c1\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") "
Sep 09 03:52:06.605647 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.605578    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-model-cache\") pod \"261cdef7-7113-464d-8c37-7eef4ef231c1\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") "
Sep 09 03:52:06.605647 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.605627    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/261cdef7-7113-464d-8c37-7eef4ef231c1-tls-certs\") pod \"261cdef7-7113-464d-8c37-7eef4ef231c1\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") "
Sep 09 03:52:06.605868 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.605680    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m89x8\" (UniqueName: \"kubernetes.io/projected/261cdef7-7113-464d-8c37-7eef4ef231c1-kube-api-access-m89x8\") pod \"261cdef7-7113-464d-8c37-7eef4ef231c1\" (UID: \"261cdef7-7113-464d-8c37-7eef4ef231c1\") "
Sep 09 03:52:06.605868 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.605786    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-home" (OuterVolumeSpecName: "home") pod "261cdef7-7113-464d-8c37-7eef4ef231c1" (UID: "261cdef7-7113-464d-8c37-7eef4ef231c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:52:06.605868 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.605846    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "261cdef7-7113-464d-8c37-7eef4ef231c1" (UID: "261cdef7-7113-464d-8c37-7eef4ef231c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:52:06.606035 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.605930    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "261cdef7-7113-464d-8c37-7eef4ef231c1" (UID: "261cdef7-7113-464d-8c37-7eef4ef231c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:52:06.606035 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.605998    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-home\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:52:06.606035 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.606017    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-tmp-dir\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:52:06.606035 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.606028    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-model-cache\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:52:06.607906 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.607881    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261cdef7-7113-464d-8c37-7eef4ef231c1-kube-api-access-m89x8" (OuterVolumeSpecName: "kube-api-access-m89x8") pod "261cdef7-7113-464d-8c37-7eef4ef231c1" (UID: "261cdef7-7113-464d-8c37-7eef4ef231c1"). InnerVolumeSpecName "kube-api-access-m89x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:52:06.607994 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.607879    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-dshm" (OuterVolumeSpecName: "dshm") pod "261cdef7-7113-464d-8c37-7eef4ef231c1" (UID: "261cdef7-7113-464d-8c37-7eef4ef231c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:52:06.607994 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.607880    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261cdef7-7113-464d-8c37-7eef4ef231c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "261cdef7-7113-464d-8c37-7eef4ef231c1" (UID: "261cdef7-7113-464d-8c37-7eef4ef231c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:52:06.706736 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.706698    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/261cdef7-7113-464d-8c37-7eef4ef231c1-dshm\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:52:06.706736 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.706731    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/261cdef7-7113-464d-8c37-7eef4ef231c1-tls-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:52:06.706935 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.706747    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m89x8\" (UniqueName: \"kubernetes.io/projected/261cdef7-7113-464d-8c37-7eef4ef231c1-kube-api-access-m89x8\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
Sep 09 03:52:06.933070 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.933032    2560 generic.go:356] "Generic (PLEG): container finished" podID="261cdef7-7113-464d-8c37-7eef4ef231c1" containerID="b85e4d1f2aa1ccf8438e52822b9189a14d6404043ad4bccae467d375d55a4f65" exitCode=0
Sep 09 03:52:06.933243 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.933077    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"
Sep 09 03:52:06.933243 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.933084    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx" event={"ID":"261cdef7-7113-464d-8c37-7eef4ef231c1","Type":"ContainerDied","Data":"b85e4d1f2aa1ccf8438e52822b9189a14d6404043ad4bccae467d375d55a4f65"}
Sep 09 03:52:06.933243 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.933118    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx" event={"ID":"261cdef7-7113-464d-8c37-7eef4ef231c1","Type":"ContainerDied","Data":"8712b121813292130a86b04096013542bcccefc3983a6ca46b48aabee6c1fa79"}
Sep 09 03:52:06.933243 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.933135    2560 scope.go:117] "RemoveContainer" containerID="b85e4d1f2aa1ccf8438e52822b9189a14d6404043ad4bccae467d375d55a4f65"
Sep 09 03:52:06.942753 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.942735    2560 scope.go:117] "RemoveContainer" containerID="b85e4d1f2aa1ccf8438e52822b9189a14d6404043ad4bccae467d375d55a4f65"
Sep 09 03:52:06.943035 ip-10-0-133-131 kubenswrapper[2560]: E0909 03:52:06.943013    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b85e4d1f2aa1ccf8438e52822b9189a14d6404043ad4bccae467d375d55a4f65\": container with ID starting with b85e4d1f2aa1ccf8438e52822b9189a14d6404043ad4bccae467d375d55a4f65 not found: ID does not exist" containerID="b85e4d1f2aa1ccf8438e52822b9189a14d6404043ad4bccae467d375d55a4f65"
Sep 09 03:52:06.943142 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.943040    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85e4d1f2aa1ccf8438e52822b9189a14d6404043ad4bccae467d375d55a4f65"} err="failed to get container status \"b85e4d1f2aa1ccf8438e52822b9189a14d6404043ad4bccae467d375d55a4f65\": rpc error: code = NotFound desc = could not find container \"b85e4d1f2aa1ccf8438e52822b9189a14d6404043ad4bccae467d375d55a4f65\": container with ID starting with b85e4d1f2aa1ccf8438e52822b9189a14d6404043ad4bccae467d375d55a4f65 not found: ID does not exist"
Sep 09 03:52:06.956343 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.956321    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"]
Sep 09 03:52:06.966075 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:06.966053    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c5827fc9-kserve-5d87556bc-pdrdx"]
Sep 09 03:52:07.358967 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:07.358935    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="261cdef7-7113-464d-8c37-7eef4ef231c1" path="/var/lib/kubelet/pods/261cdef7-7113-464d-8c37-7eef4ef231c1/volumes"
Sep 09 03:52:13.301287 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:13.301250    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:52:20.293875 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:20.293839    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:52:26.088684 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:26.088646    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:52:32.806907 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:32.806875    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fdtxc"]
Sep 09 03:52:35.995523 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:35.995486    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-857c4ff9cb-c76z4_36e730bb-c479-4b26-b925-30ac5137dcd3/authorino/0.log"
Sep 09 03:52:42.442057 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:42.442028    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-ff49c877-vsrxv_4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624/copy-manifests/0.log"
Sep 09 03:52:42.449747 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:42.449727    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-ff49c877-vsrxv_4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624/manager/0.log"
Sep 09 03:52:44.234046 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:44.234001    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-857c4ff9cb-c76z4_36e730bb-c479-4b26-b925-30ac5137dcd3/authorino/0.log"
Sep 09 03:52:44.578827 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:44.578747    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-ff2jp_c6a6bf66-e3a4-46da-8a98-b60501b8fded/kuadrant-console-plugin/0.log"
Sep 09 03:52:44.820006 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:44.819974    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-fdtxc_eed51b4a-fc6b-40a8-ae0c-d5ae12f45a34/limitador/0.log"
Sep 09 03:52:45.518460 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:45.518428    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58f995d999-7gwbh_97abd95b-8739-4ed4-bfd4-397a25c27de0/kube-auth-proxy/0.log"
Sep 09 03:52:46.705881 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:46.705853    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-g8mt6_fa26adec-42ee-4838-b48c-4cdda3fc6a60/main/0.log"
Sep 09 03:52:46.947090 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:46.947062    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fssjgq_7c0884da-95f2-4338-ae6b-ffbf5b9d7267/main/0.log"
Sep 09 03:52:53.172151 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.172114    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qghgg/must-gather-rgt2l"]
Sep 09 03:52:53.172633 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.172483    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261cdef7-7113-464d-8c37-7eef4ef231c1" containerName="main"
Sep 09 03:52:53.172633 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.172493    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="261cdef7-7113-464d-8c37-7eef4ef231c1" containerName="main"
Sep 09 03:52:53.172633 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.172550    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="261cdef7-7113-464d-8c37-7eef4ef231c1" containerName="main"
Sep 09 03:52:53.175708 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.175692    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qghgg/must-gather-rgt2l"
Sep 09 03:52:53.180385 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.180365    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qghgg\"/\"default-dockercfg-cw55d\""
Sep 09 03:52:53.181200 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.181186    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qghgg\"/\"kube-root-ca.crt\""
Sep 09 03:52:53.182039 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.181838    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qghgg\"/\"openshift-service-ca.crt\""
Sep 09 03:52:53.197072 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.197052    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qghgg/must-gather-rgt2l"]
Sep 09 03:52:53.199383 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.199357    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:52:53.223818 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.223795    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dd7k\" (UniqueName: \"kubernetes.io/projected/75b5e190-9492-43a2-bb54-57c54d1f2e99-kube-api-access-7dd7k\") pod \"must-gather-rgt2l\" (UID: \"75b5e190-9492-43a2-bb54-57c54d1f2e99\") " pod="openshift-must-gather-qghgg/must-gather-rgt2l"
Sep 09 03:52:53.223921 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.223846    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/75b5e190-9492-43a2-bb54-57c54d1f2e99-must-gather-output\") pod \"must-gather-rgt2l\" (UID: \"75b5e190-9492-43a2-bb54-57c54d1f2e99\") " pod="openshift-must-gather-qghgg/must-gather-rgt2l"
Sep 09 03:52:53.324530 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.324499    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/75b5e190-9492-43a2-bb54-57c54d1f2e99-must-gather-output\") pod \"must-gather-rgt2l\" (UID: \"75b5e190-9492-43a2-bb54-57c54d1f2e99\") " pod="openshift-must-gather-qghgg/must-gather-rgt2l"
Sep 09 03:52:53.324682 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.324636    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dd7k\" (UniqueName: \"kubernetes.io/projected/75b5e190-9492-43a2-bb54-57c54d1f2e99-kube-api-access-7dd7k\") pod \"must-gather-rgt2l\" (UID: \"75b5e190-9492-43a2-bb54-57c54d1f2e99\") " pod="openshift-must-gather-qghgg/must-gather-rgt2l"
Sep 09 03:52:53.324875 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.324858    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/75b5e190-9492-43a2-bb54-57c54d1f2e99-must-gather-output\") pod \"must-gather-rgt2l\" (UID: \"75b5e190-9492-43a2-bb54-57c54d1f2e99\") " pod="openshift-must-gather-qghgg/must-gather-rgt2l"
Sep 09 03:52:53.335201 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.335178    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dd7k\" (UniqueName: \"kubernetes.io/projected/75b5e190-9492-43a2-bb54-57c54d1f2e99-kube-api-access-7dd7k\") pod \"must-gather-rgt2l\" (UID: \"75b5e190-9492-43a2-bb54-57c54d1f2e99\") " pod="openshift-must-gather-qghgg/must-gather-rgt2l"
Sep 09 03:52:53.356690 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.356661    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log" limit=1024
Sep 09 03:52:53.484099 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.484022    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qghgg/must-gather-rgt2l"
Sep 09 03:52:53.768279 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.768254    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qghgg/must-gather-rgt2l"]
Sep 09 03:52:53.770733 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:52:53.770700    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75b5e190_9492_43a2_bb54_57c54d1f2e99.slice/crio-97cb02c8d4f9ffa7569cf504e2aac4de0b6ec2fd5d9c803420d87f9dab58158c WatchSource:0}: Error finding container 97cb02c8d4f9ffa7569cf504e2aac4de0b6ec2fd5d9c803420d87f9dab58158c: Status 404 returned error can't find the container with id 97cb02c8d4f9ffa7569cf504e2aac4de0b6ec2fd5d9c803420d87f9dab58158c
Sep 09 03:52:53.772739 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:53.772720    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:52:54.158328 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:54.158295    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qghgg/must-gather-rgt2l" event={"ID":"75b5e190-9492-43a2-bb54-57c54d1f2e99","Type":"ContainerStarted","Data":"97cb02c8d4f9ffa7569cf504e2aac4de0b6ec2fd5d9c803420d87f9dab58158c"}
Sep 09 03:52:55.168809 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:55.168068    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qghgg/must-gather-rgt2l" event={"ID":"75b5e190-9492-43a2-bb54-57c54d1f2e99","Type":"ContainerStarted","Data":"3a5fe8a4ad72ff198900c194263b5686ea5d2573586cd11e7d11318e97862cd2"}
Sep 09 03:52:55.168809 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:55.168126    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qghgg/must-gather-rgt2l" event={"ID":"75b5e190-9492-43a2-bb54-57c54d1f2e99","Type":"ContainerStarted","Data":"f908e7df97813b042f440f47e185f1d3c1c44eb55949204cadb6a8e555c2c05d"}
Sep 09 03:52:55.189417 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:55.189353    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qghgg/must-gather-rgt2l" podStartSLOduration=1.148804162 podStartE2EDuration="2.18932948s" totalImagesPullingTime="1.040525318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:52:55.187282926 +0000 UTC m=+2392.324030792" watchObservedRunningTime="2026-09-09 03:52:55.18932948 +0000 UTC m=+2392.326077349"
Sep 09 03:52:56.622660 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:56.622625    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v89sv_422df607-4fd9-4edd-ad99-7b6214c7d970/global-pull-secret-syncer/0.log"
Sep 09 03:52:56.695240 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:56.695201    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7xspc_8465770f-1ae2-4566-8c6d-12cc331da1ec/konnectivity-agent/0.log"
Sep 09 03:52:56.826093 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:52:56.826062    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-131.ec2.internal_50ae6cedb6cbe5050a1216b96fc89c4f/haproxy/0.log"
Sep 09 03:53:01.147255 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:01.146575    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-857c4ff9cb-c76z4_36e730bb-c479-4b26-b925-30ac5137dcd3/authorino/0.log"
Sep 09 03:53:01.328404 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:01.328314    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-ff2jp_c6a6bf66-e3a4-46da-8a98-b60501b8fded/kuadrant-console-plugin/0.log"
Sep 09 03:53:01.433870 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:01.433798    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-fdtxc_eed51b4a-fc6b-40a8-ae0c-d5ae12f45a34/limitador/0.log"
Sep 09 03:53:03.057425 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:03.057383    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0fa800a9-04b3-49d9-8748-89b7afb841b8/alertmanager/0.log"
Sep 09 03:53:03.091413 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:03.091378    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0fa800a9-04b3-49d9-8748-89b7afb841b8/config-reloader/0.log"
Sep 09 03:53:03.121738 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:03.121695    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0fa800a9-04b3-49d9-8748-89b7afb841b8/kube-rbac-proxy-web/0.log"
Sep 09 03:53:03.165128 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:03.165048    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0fa800a9-04b3-49d9-8748-89b7afb841b8/kube-rbac-proxy/0.log"
Sep 09 03:53:03.194195 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:03.194156    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0fa800a9-04b3-49d9-8748-89b7afb841b8/kube-rbac-proxy-metric/0.log"
Sep 09 03:53:03.227292 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:03.227182    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0fa800a9-04b3-49d9-8748-89b7afb841b8/prom-label-proxy/0.log"
Sep 09 03:53:03.269959 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:03.269924    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0fa800a9-04b3-49d9-8748-89b7afb841b8/init-config-reloader/0.log"
Sep 09 03:53:03.339979 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:03.338821    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-snfbl_aec25042-0f8f-4bda-a8bf-f1adb30990a6/cluster-monitoring-operator/0.log"
Sep 09 03:53:03.417397 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:03.417359    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-qs94b_b8c76543-c8a2-4f13-891a-127f0afdb670/kube-state-metrics/0.log"
Sep 09 03:53:03.475228 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:03.475201    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-qs94b_b8c76543-c8a2-4f13-891a-127f0afdb670/kube-rbac-proxy-main/0.log"
Sep 09 03:53:03.519007 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:03.518976    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-qs94b_b8c76543-c8a2-4f13-891a-127f0afdb670/kube-rbac-proxy-self/0.log"
Sep 09 03:53:03.907235 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:03.907204    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wq9cn_bb1b3371-60f4-4ef6-ae93-f55d583b2eb6/node-exporter/0.log"
Sep 09 03:53:03.990007 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:03.989921    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wq9cn_bb1b3371-60f4-4ef6-ae93-f55d583b2eb6/kube-rbac-proxy/0.log"
Sep 09 03:53:04.105737 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:04.105705    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wq9cn_bb1b3371-60f4-4ef6-ae93-f55d583b2eb6/init-textfile/0.log"
Sep 09 03:53:04.388041 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:04.388007    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_334da5a2-1482-4597-8a4b-18b11c9cf3dd/prometheus/0.log"
Sep 09 03:53:04.433283 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:04.433249    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_334da5a2-1482-4597-8a4b-18b11c9cf3dd/config-reloader/0.log"
Sep 09 03:53:04.464232 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:04.464198    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_334da5a2-1482-4597-8a4b-18b11c9cf3dd/thanos-sidecar/0.log"
Sep 09 03:53:04.508083 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:04.508049    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_334da5a2-1482-4597-8a4b-18b11c9cf3dd/kube-rbac-proxy-web/0.log"
Sep 09 03:53:04.541793 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:04.541761    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_334da5a2-1482-4597-8a4b-18b11c9cf3dd/kube-rbac-proxy/0.log"
Sep 09 03:53:04.582475 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:04.582441    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_334da5a2-1482-4597-8a4b-18b11c9cf3dd/kube-rbac-proxy-thanos/0.log"
Sep 09 03:53:04.612666 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:04.612635    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_334da5a2-1482-4597-8a4b-18b11c9cf3dd/init-config-reloader/0.log"
Sep 09 03:53:04.652986 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:04.652901    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-hcs7l_a94c8e5a-97f8-4a15-bf3d-b73758df0235/prometheus-operator/0.log"
Sep 09 03:53:04.690687 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:04.690655    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-hcs7l_a94c8e5a-97f8-4a15-bf3d-b73758df0235/kube-rbac-proxy/0.log"
Sep 09 03:53:04.726274 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:04.726242    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-hbmnj_3536d6ff-720b-4bf3-b4c6-c4f75a2cffeb/prometheus-operator-admission-webhook/0.log"
Sep 09 03:53:06.291913 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.291879    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-5jzgn_6b8b2e70-6c05-4bfb-98ce-e18701db3e63/networking-console-plugin/0.log"
Sep 09 03:53:06.773718 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.773681    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"]
Sep 09 03:53:06.779387 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.779360    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:06.790931 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.790905    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"]
Sep 09 03:53:06.876289 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.876251    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91433392-5184-4a68-9596-5477e6461579-sys\") pod \"perf-node-gather-daemonset-w24v4\" (UID: \"91433392-5184-4a68-9596-5477e6461579\") " pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:06.876473 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.876339    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91433392-5184-4a68-9596-5477e6461579-lib-modules\") pod \"perf-node-gather-daemonset-w24v4\" (UID: \"91433392-5184-4a68-9596-5477e6461579\") " pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:06.876473 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.876361    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/91433392-5184-4a68-9596-5477e6461579-podres\") pod \"perf-node-gather-daemonset-w24v4\" (UID: \"91433392-5184-4a68-9596-5477e6461579\") " pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:06.876473 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.876378    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlgrl\" (UniqueName: \"kubernetes.io/projected/91433392-5184-4a68-9596-5477e6461579-kube-api-access-wlgrl\") pod \"perf-node-gather-daemonset-w24v4\" (UID: \"91433392-5184-4a68-9596-5477e6461579\") " pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:06.876678 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.876488    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/91433392-5184-4a68-9596-5477e6461579-proc\") pod \"perf-node-gather-daemonset-w24v4\" (UID: \"91433392-5184-4a68-9596-5477e6461579\") " pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:06.977814 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.977769    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91433392-5184-4a68-9596-5477e6461579-sys\") pod \"perf-node-gather-daemonset-w24v4\" (UID: \"91433392-5184-4a68-9596-5477e6461579\") " pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:06.978002 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.977869    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91433392-5184-4a68-9596-5477e6461579-lib-modules\") pod \"perf-node-gather-daemonset-w24v4\" (UID: \"91433392-5184-4a68-9596-5477e6461579\") " pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:06.978002 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.977897    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/91433392-5184-4a68-9596-5477e6461579-podres\") pod \"perf-node-gather-daemonset-w24v4\" (UID: \"91433392-5184-4a68-9596-5477e6461579\") " pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:06.978002 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.977925    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlgrl\" (UniqueName: \"kubernetes.io/projected/91433392-5184-4a68-9596-5477e6461579-kube-api-access-wlgrl\") pod \"perf-node-gather-daemonset-w24v4\" (UID: \"91433392-5184-4a68-9596-5477e6461579\") " pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:06.978002 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.977968    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/91433392-5184-4a68-9596-5477e6461579-proc\") pod \"perf-node-gather-daemonset-w24v4\" (UID: \"91433392-5184-4a68-9596-5477e6461579\") " pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:06.978238 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.978112    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/91433392-5184-4a68-9596-5477e6461579-proc\") pod \"perf-node-gather-daemonset-w24v4\" (UID: \"91433392-5184-4a68-9596-5477e6461579\") " pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:06.978238 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.978176    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91433392-5184-4a68-9596-5477e6461579-sys\") pod \"perf-node-gather-daemonset-w24v4\" (UID: \"91433392-5184-4a68-9596-5477e6461579\") " pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:06.978370 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.978291    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91433392-5184-4a68-9596-5477e6461579-lib-modules\") pod \"perf-node-gather-daemonset-w24v4\" (UID: \"91433392-5184-4a68-9596-5477e6461579\") " pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:06.978436 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.978369    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/91433392-5184-4a68-9596-5477e6461579-podres\") pod \"perf-node-gather-daemonset-w24v4\" (UID: \"91433392-5184-4a68-9596-5477e6461579\") " pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:06.988715 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:06.988682    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlgrl\" (UniqueName: \"kubernetes.io/projected/91433392-5184-4a68-9596-5477e6461579-kube-api-access-wlgrl\") pod \"perf-node-gather-daemonset-w24v4\" (UID: \"91433392-5184-4a68-9596-5477e6461579\") " pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:07.015790 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:07.015743    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:53:07.026425 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:07.026340    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/2.log"
Sep 09 03:53:07.093021 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:07.092983    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:07.406453 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:07.406408    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"]
Sep 09 03:53:07.408003 ip-10-0-133-131 kubenswrapper[2560]: W0909 03:53:07.407974    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod91433392_5184_4a68_9596_5477e6461579.slice/crio-5104fb919be313a9cc4116196e4177d5a8b84b584f9151324832c77a36d2e634 WatchSource:0}: Error finding container 5104fb919be313a9cc4116196e4177d5a8b84b584f9151324832c77a36d2e634: Status 404 returned error can't find the container with id 5104fb919be313a9cc4116196e4177d5a8b84b584f9151324832c77a36d2e634
Sep 09 03:53:07.653884 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:07.653790    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-69f9b5898b-92slb_6780a37d-e6d8-432b-b32c-9082c87315f5/console/0.log"
Sep 09 03:53:07.698118 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:07.698087    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-k6ndc_108cfd1b-5044-402e-8828-67b05550857a/download-server/0.log"
Sep 09 03:53:08.264165 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:08.264121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4" event={"ID":"91433392-5184-4a68-9596-5477e6461579","Type":"ContainerStarted","Data":"a850dc74fb3c5075cefada0ac45fdbe66ee2942e7dc61fb7a70d4dc81be10231"}
Sep 09 03:53:08.264363 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:08.264193    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4" event={"ID":"91433392-5184-4a68-9596-5477e6461579","Type":"ContainerStarted","Data":"5104fb919be313a9cc4116196e4177d5a8b84b584f9151324832c77a36d2e634"}
Sep 09 03:53:08.265029 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:08.265005    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:08.295068 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:08.295012    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4" podStartSLOduration=2.294996739 podStartE2EDuration="2.294996739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:53:08.293369714 +0000 UTC m=+2405.430117584" watchObservedRunningTime="2026-09-09 03:53:08.294996739 +0000 UTC m=+2405.431744604"
Sep 09 03:53:08.410260 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:08.410227    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-7x7k4_fa341cd0-a17e-4677-978d-3a3abcde5ad3/volume-data-source-validator/0.log"
Sep 09 03:53:09.576745 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:09.576718    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:53:09.599223 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:09.599196    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-vhcbz_5ebf6c83-e589-4833-994e-25ebfbb0b0a1/console-operator/1.log"
Sep 09 03:53:09.685083 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:09.685055    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c465c_a65c3985-522f-46f4-aeb2-7f12ae347816/dns/0.log"
Sep 09 03:53:09.715717 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:09.715689    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c465c_a65c3985-522f-46f4-aeb2-7f12ae347816/kube-rbac-proxy/0.log"
Sep 09 03:53:09.813862 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:09.813836    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sbmb7_def2fdc5-08ed-43a3-b9c1-b8b55e6444c5/dns-node-resolver/0.log"
Sep 09 03:53:10.537407 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:10.537377    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v47sj_0f4c4cf1-bf9c-499d-b25e-fe93588e6c58/node-ca/0.log"
Sep 09 03:53:11.928332 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:11.928299    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58f995d999-7gwbh_97abd95b-8739-4ed4-bfd4-397a25c27de0/kube-auth-proxy/0.log"
Sep 09 03:53:13.160131 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:13.160093    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h6lb8_90fe6e96-2fae-4cc7-97fb-1d46727849ed/serve-healthcheck-canary/0.log"
Sep 09 03:53:13.675442 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:13.675404    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-jtgvc_6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593/insights-operator/0.log"
Sep 09 03:53:13.676000 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:13.675979    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-jtgvc_6fea3e4f-7ff0-4f2d-8d8f-9dfda7ae8593/insights-operator/1.log"
Sep 09 03:53:13.712316 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:13.712284    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-85tbf_264d64b1-2bf7-4fea-acdd-0381a36e2e0c/kube-rbac-proxy/0.log"
Sep 09 03:53:13.754170 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:13.754143    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-85tbf_264d64b1-2bf7-4fea-acdd-0381a36e2e0c/exporter/0.log"
Sep 09 03:53:13.791417 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:13.791388    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-85tbf_264d64b1-2bf7-4fea-acdd-0381a36e2e0c/extractor/0.log"
Sep 09 03:53:15.296757 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:15.296728    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qghgg/perf-node-gather-daemonset-w24v4"
Sep 09 03:53:16.750720 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:16.750684    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-ff49c877-vsrxv_4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624/manager/0.log"
Sep 09 03:53:16.782609 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:16.782568    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-ff49c877-vsrxv_4cfb4c1a-4eb4-4dbb-ad2c-b4182f555624/copy-manifests/0.log"
Sep 09 03:53:17.234032 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:17.234000    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58f995d999-7gwbh_97abd95b-8739-4ed4-bfd4-397a25c27de0/kube-auth-proxy/0.log"
Sep 09 03:53:18.421279 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:18.421251    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6f5d5c9c64-7pmsk_0d783a2f-7f25-4bdc-b2b4-0d331b80ad6b/manager/0.log"
Sep 09 03:53:18.481049 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:18.481008    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-vkwbv_819bbf51-7f42-46c3-8263-e2aefdc7cc3c/openshift-lws-operator/0.log"
Sep 09 03:53:19.303895 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:19.303860    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58f995d999-7gwbh_97abd95b-8739-4ed4-bfd4-397a25c27de0/kube-auth-proxy/0.log"
Sep 09 03:53:24.163957 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:24.163922    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-dhtsn_9d8945e0-9f39-4523-b066-5125ad017318/kube-storage-version-migrator-operator/1.log"
Sep 09 03:53:24.165718 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:24.165679    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-dhtsn_9d8945e0-9f39-4523-b066-5125ad017318/kube-storage-version-migrator-operator/0.log"
Sep 09 03:53:25.635229 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:25.635202    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sbbhj_4b3084e8-6418-4139-9623-5528c6b2cef3/kube-multus-additional-cni-plugins/0.log"
Sep 09 03:53:25.663844 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:25.663764    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sbbhj_4b3084e8-6418-4139-9623-5528c6b2cef3/egress-router-binary-copy/0.log"
Sep 09 03:53:25.710463 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:25.710423    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sbbhj_4b3084e8-6418-4139-9623-5528c6b2cef3/cni-plugins/0.log"
Sep 09 03:53:25.736273 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:25.736239    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sbbhj_4b3084e8-6418-4139-9623-5528c6b2cef3/bond-cni-plugin/0.log"
Sep 09 03:53:25.764092 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:25.764061    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sbbhj_4b3084e8-6418-4139-9623-5528c6b2cef3/routeoverride-cni/0.log"
Sep 09 03:53:25.794715 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:25.794670    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sbbhj_4b3084e8-6418-4139-9623-5528c6b2cef3/whereabouts-cni-bincopy/0.log"
Sep 09 03:53:25.824848 ip-10-0-133-131 kubenswrapper[2560]: I0909 03:53:25.824818    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sbbhj_4b3084e8-6418-4139-9623-5528c6b2cef3/whereabouts-cni/0.log"